Messaging mechanism for inter processor communication

ABSTRACT

An apparatus and method are provided for connecting a host Enterprise System Connection Architecture (ESCON) Input/Output (I/O) interface to a cache of a data storage system. The apparatus includes (a) a set of at least 4 pipelines, each pipeline being coupled on a first end to the host ESCON I/O interface and being coupled on a second end to the cache, (b) a plurality of line processors, each line processor controlling one or more of the pipelines of the set of pipelines, and (c) in each pipeline, a protocol engine, the protocol engine configured to distinguish user data from frame header data and separate the user data from the frame header data for transport over the pipeline.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.09/213,613 entitled “MESSAGING MECHANISM EMPLOYING MAILBOXES FOR INTERPROCESSOR COMMUNICATIONS,” filed on Dec. 18, 1998, the teachings ofwhich are hereby incorporated by reference in their entirety.

FIELD OF THE INVENTION

The present invention relates to storage devices, and more particularlyto adapters for interfacing a storage device to a host for high speed,high reliability and high availability data storage.

BACKGROUND OF INVENTION

Various interfaces are known for interconnecting host computer systemsto storage systems, such as disk drive mass storage systems. Particularinterfaces involve respective particular data transfer protocols andsystem interconnections. The IBM Enterprise System ConnectionArchitecture (ESCON) is a flexible interface or interconnectionenvironment that is used to move data from a host source to storage andback. ESCON, as it is known in the art, combines technology andarchitecture to include fiber optic cabling for transmission andreception of data. ESCON implements dynamic connectivity throughswitched point to point topology and data flow interconnectivity withother networks. The ESCON I/O architecture exploits the fiber optictechnology implementation and the concept of dynamic connectivity inproviding a flexible, extensible interface.

The ESCON architecture has become a widely accepted standard for datacommunications in large scale computer environments, replacing the moretraditional IBM “Bus and Tag” protocol. ESCON employs serially encodeddata transmission techniques in place of parallel data protocols.

The ESCON architecture provides inherent advantages such as: informationtransfer using fiber optic technology; higher rates of data transfer(17MBytes/sec); extended distances (3 km typical with extension to 60km); dynamic connectivity through switched point-to-point data flow; andinterconnectivity with local and wide area networks

Application of ESCON as the means for data communications can, inaddition to the asserted enhancements in data throughput rates anddistance, provide other meaningful benefits such as: greater connectionflexibility of equipment; reduced floor loading due to significant cableweight and size reductions; customer configuration expansion orreconfiguration with minimal or no disruption; increased data integrityand security; and reduced cost of ownership through more effectiveutilization of equipment.

The ESCON architecture has been adopted by many other manufacturers ofcomputer equipment as a basic input/output protocol and it has beenaccepted as an ANSI Standard (“SBCON”). The technical details of the IBMESCON interface are described, among other places, in various IBMpublications including INTRODUCING ENTERPRISE SYSTEMS CONNECTION, IBM3990 ESCON FUNCTION, INSTALLATION AND MIGRATION, IBM 3990 STORAGECONTROL ESCON FEATURES PRESENTATION GUIDE, ENTERPRISE SYSTEMSARCHITECTURE/390 ESCON I/O INTERFACE AND ENTERPRISE SYSTEMSARCHITECTURE, which are incorporated herein by reference.

The various devices interconnected to a host system with ESCON I/Oinclude storage systems known as “Integrated Cached Disk Arrays,”(“ICDAs”), which are typically an array of small inexpensive disk drivesintegrated into a single chassis. High speed caches are implementedbetween the host and disks in ICDAs to yield improved performance. Onefamily of known ICDA products, known as SYMMETRIX produced by EMCCorporation, Hopkinton, Mass., provides a high reliability array ofdrives, and offers great flexibility in terms of performanceenhancements such as: mirroring; greater data availability; greater datatransfer rates over distributed buses; and various levels of redundancyimplemented in systems referred to as “RAID systems” (“Redundant Arraysof Inexpensive Disks”).

The EMC² Symmetrix architecture, generally illustrated in FIG. 1,integrates a high speed cache or global memory between a disk array anda host computer or CPU. The functional elements generally required tointegrate the cache include a host-to-cache interface (which in oneimplementation is an IBM standard ESCON interface referred to as a host“ESCON Adapter”—EA), and a cache-to-disk drives interface (which may bea Small Computer Systems Interface, “SCSI”, referred to as a “DiskAdapter”—DA). The EA and DA interface boards are generically referred toas “Directors”. The Symmetrix architecture operates under a “cache all”policy, meaning that all transfers, i.e. from the host to the drives orfrom the drives to the host, go through cache. The principal function ofthe Directors is the movement of data between the host and Global Memory(cache) or between Global Memory and the Disk Drives.

The Global Memory Bus (GMB), between the EA and cache and between the DAand cache in Symmetrix, actually consists of two portions or identicalbuses designated “A” and “B”. The use of two buses improves performanceand eliminates a possible single point of failure. Each bus hasindependent arbitration and consists of a 32 bit address bus plus 1parity, a 64 bit data bus with 8 bits of Error Correction Code (ECC)check bits and a number of control lines with parity. The smallest datatransfer that may take place over the GMB is 64 bits during each access(however, byte, word and longword operations are performed within aDirector). The SYMMETRIX family of ICDAs are described in detail in theSymmetrix Product Manuals (for Models 5500, 52XX, 5100, 3500, 32XX and3100) which are incorporated herein by reference.

The ESCON adapter interface(s) and disk adapter interface(s), i.e. thebus(es) between the host and EAs and between the disk array and DAsrespectively, are 8 bit and in some cases 16 bit interfaces inSymmetrix. Thus the bytes received from a host have to be assembled intoa 64-bit memory word for transfer to Global Memory, i.e. cache.Similarly, 64-bit memory words from Global Memory have to bedisassembled into bytes for transmission over the interface(s).

Assembly/disassembly of data to/from Global Memory is carried out byplural gate arrays located on each Director.

The Symmetrix family of ICDAs is designed around a pipelinedarchitecture. A pipeline or pipe in the system is a registered pathalong which data is clocked to move it from one location to another.

The Directors, generally, are designed around a common pipelinedarchitecture moving data along the “pipe” or pipeline undermicroprocessor control. It is the pipelines that move the data when thesystem is operating, not the controlling microprocessors. Themicroprocessors set-up the pipes to perform a transfer and monitor thepipelines for errors during the transfer. The Directors in knownSymmetrix systems incorporate a dual control processor architectureincluding a first processor referred to as “X” and a second processorreferred to as “Y”. The dual processor architecture is configured toshare substantial resources in order to keep hardware requirements to aminimum. Each control processor in a “front end” Director, i.e.transferring data to/from Global Memory, is typically responsible fortwo pipelines designated “A” and “B” and respective Direct MultipleAccess (DMA) and Direct Single Access (DSA) pipelines, for Global Memoryaccess.

Known Symmetrix systems can be configured with ESCON adapter circuitryfor communication with hosts having ESCON I/O, as disclosed in U.S.patent application Ser. No. 08/753,673 which is incorporated herein byreference. FIG. 2 illustrates a known ESCON front end Adapter orDirector, configured for transferring data between a host ESCON I/O (notshown) according to the IBM Bus ESCON standard known in the art and theICDA incorporating the Global Memory. Data in accordance with the ESCONprotocol is received on the ESCON Director by ESCON Receiver/TransmitterGate Arrays (Rx/Tx GA) which receive and transmit ESCON data between thehost ESCON I/O and the ESCON Director pipes in accord with the ESCON I/Oprotocol. ESCON data is transmitted in two physical pipes that areconfigured to operate as four pipes, two on each side: an A and B pipe(XA and XB for transmit and receive, respectively) on the X sidecontrolled by an X control processor, and an A and B pipe (YA and YB fortransmit and receive, respectively) on the Y side controlled by a Ycontrol processor. Parallel ESCON data is passed in the pipes on theESCON front end Director on a bidirectional bus connected to respectiveESCON Receiver/Transmitter Gate Arrays (Rx/Tx GA), which in turnassemble/disassemble 64 bit memory words. Error detection and correctionis performed in the ESCON pipe using a pass through EDAC device. DualPort Ram (DPR) is implemented in the pipe for buffering data transferredbetween the ESCON host (not shown) and Global Memory.

Each processor on the ESCON Director has its own Memory Data Register(MDR) to support DMA/DSA activity. The MDR provides a 72 bit wideregister set, 64 bit data and 8 bits parity, comprised of upper andlower words that can be independently read or written by the respectivemicroprocessor. Similarly, the MDR performs the assembly and disassemblyof 64 bit Global Memory words.

The two microprocessors on the known ESCON Director perform functionsassociated with managing data flow to and from Global Memory, andadditionally are accessible as individual points of contact for a userto configure memory and/or test and exercise the memory and pipelines(e.g. for diagnostic purposes). Such an implementation candisadvantageously interfere with the data flow management and addscomplexity to the system. Complexity is inherent in the configuration(s)known in the art, in that the individual pipeline or line processor'sactivities, beyond data flow management, must be coordinated with otherline processors' activities. Furthermore, involving line processors inother than data flow management leads to bottlenecks on the backplanebus that negatively affect system performance. For example,inter-processor communication, i.e. communication between lineprocessors, interferes with the system's data flow capabilities.

Known ICDA implementations, such as known Symmetrix systems, aredesigned according to a two level architecture. That is, the Directorinterface is a first level interface that works in conjunction with asecond level adapter board that is directly connected to the host ESCONI/O. The ESCON Receiver/Transmitter Gate Arrays (Rx/Tx GA) on theDirector transmit data to or receive data from the second level ESCONadapter board that effects data conversion using devices known in theart. The second level adapter board converts parallel data to serialESCON data using a Cypress CY7B923 device (for transmission to thehost), or converts serial ESCON data to parallel with a Cypress CY7B933(for receipt by the ESCON Director). A processor on the second leveladapter board handles protocol operations and basically translates ESCONinformation into a generic interface protocol, or interchange format.The adapter board includes memory associated with the adapter processor,and all of the data from the ESCON protocol engines is storedtemporarily in this memory. The processor on the adapter then has tofigure out by looking through the stored data where header informationis and where the data is. The adapter processor then writes in theadapter memory information about where the data starts and ends. Inaddition, the processor interprets commands from the host and writesinto the adapter memory what the command is and what the ESCON commandmeans in a generic fashion.

The Director card processor then finds this information. The processoron the Director(s) polls through the adapter memory space to finddirectives from the adapter. For example, an ESCON Director polling theadapter memory could find a write operation along with an indication ofwhere the data to be written is located. The converted ESCON informationfound can then be handled by the processor(s) on the Director(s), whichtake the data and actually do transfers to/from the Global Memory.

Such a two level architecture results in significant loss ofperformance, in part because of the polling for the data transferinformation. With the two levels, the adapter card at one level has tofigure out what the command is, where the data is, translate it to theinterchange format and store all this in the adapter memory. At thesecond level, the Director polls and retrieves the interchangeinformation. There is very little pipelining that can be effected insuch an implementation. Inefficiencies result from the architecturewherein operations are completed at one level and handed off to anotherlevel. Parallelism is not exploited.

Similarly, known ICDA implementations, and other arrays of inexpensivedisks that transfer data according to the known ESCON protocol, receiveuser data for disk storage in the context of a frame that includesheader information and frame information. Known integrated cache systemstypically store the data in the cache memory along with the header andframe information. Such an implementation is not an efficient use of thecache memory capacity. Disadvantageously, complex software has to beconfigured to effectively track the location of the data in memory amongESCON header/frame information. Movement of cached data, i.e. going toor from the disk array, in known systems requires that the data bediscriminated from extraneous cached protocol information whichnegatively affects transmission speed and efficiency.

Additionally, storage of user data among header/frame informationincreases the chances of conflicts arising between line processorsaccessing the cache to retrieve user data and other line processorsperusing the cache to ascertain pertinent header/frame information.Complex pointer management is required to maintain coherency betweenmultiple processors accessing cache storing data and header/frameinformation.

SUMMARY OF INVENTION

The present invention provides an ESCON interface for a high performanceintegrated cache storage device implemented in an architecture having ahigh degree of parallelism and a single level of protocol conversion.

According to the invention, the ESCON interface is implemented as aninterface between a host ESCON I/O and global memory and has anarchitecture that includes a plurality of pipelines each of which iscontrolled by a respective line processor. In one implementation, twopipelines are controlled by one processor and the interface is flexiblyconfigured, under software control, to implement four or eightpipelines. Each pipeline receives/transmits ESCON I/O information over arespective fiber optic transceiver. An onboard ESCON protocol conversiondevice or protocol engine in each pipe is configured with the capabilityof distinguishing data which is customer, or user, data that is to bestored on a disk or is read from disk versus the headers and informationthat defines control frames in which the data is presented. Transmit andreceive frame dual port rams (“xmit frame DPR” and “receive frame DPR”,respectively), for storing transmitted frame and received frameinformation, are connected to the protocol engines which strip theframe/header information from the user data. The control information isput into a format accessible via ring pointers and into a structure,i.e. the DPR, that is easily manipulated using the onboard lineprocessor. The user data that is to be stored in Global Memory (“GM”)separated from the control information that comes from the host computeris stored temporarily in First In First Out registers (FIFOs). Thus, thearchitecture according to the present invention implements hardware thateffects discrimination between header/frame information and user data,and effects stripping and storage of header/frame information apart fromuser data processed for storage in cache.

In further accord with the invention, user data processed for storage incache, which is effectively concatenated by stripping the frame/headerinformation and storing the separated user data in the FIFOs, issubjected to enhanced error detection. The protocol engine is configuredto compute Cyclic Redundancy Check (CRC) error detection information onthe separated (concatenated) user data after stripping the frame/headerinformation. Thus although any CRC data transferred from the host withthe data frame is stripped, CRC information is computed for theseparated data at a point close to the physical interface delivering thedata to the ESCON interface according to the invention. The integrity ofthe data can therefore be tested again as the data moves through thepipeline(s).

Additionally, data integrity is further ensured by checking theintegrity of the simplified pointer management implemented upon enteringthe user data in the FIFOs. The order, and thus integrity, of datamoving through the FIFOs is checked by an enhanced parity mechanism. Theuser data is stored as bytes in the FIFOs which are nine bits wide tofacilitate a parity bit, as known in the art. This parity checkmechanism is enhanced by a pseudo-random number generated in a linearfeedback shift register (LFSR) in the protocol engine field programmablegate array (FPGA). The pseudo-random number is used to determine whetherthe parity should be odd or even for that byte when the data bytes exitthe FIFOs.

An assembler/disassembler mechanism in each pipeline according to theinvention receives data from FIFOs (on a write operation to GM), andtransfers data to FIFOs (on a read operation from GM). A substantiallyidentical enhanced parity mechanism as implemented in the protocolengine is implemented in the assembler/disassembler to generate apseudo-random number for each data byte. The enhanced parityaccompanying data transferred from the FIFO (on a write to GM) isexclusive-ORed with LFSR data generated by enhanced parity circuitry inthe assembler/disassembler which effectively reproduces the originalparity for the data. If the pointers in the FIFOs become disjoined sothat data is essentially being read out-of-sequence from the FIFOs, thecondition will be detected in a significant number of the cases as itshows up as a parity error. Thus addressing errors in the FIFOs willmanifest themselves as parity errors if the data is stored in the FIFOswith opposite parity than what is detected at the assembler/disassembler(on a write to GM), or vice versa (on a read from GM). Data transferredfrom the assembler/disassembler to the FIFO (on a read from GM) isprocessed using the enhanced parity mechanisms in substantially the samemanner.

The assembler/disassembler mechanism according to the invention is morefeature rich than known assembler/disassemblers such as described in thereferenced U.S. patent application. The present assembler/disassemblerincorporates registers that provide a pathway for the respective lineprocessor to get access into the data string, and allow the processor toaccess the GM, effectively replacing the known MDR pipeline.

The assembler/disassembler also contains a counter mechanism thatcounts/controls the amount of data that is read into and out of theFIFOs. When the data transfer starts, the line processor can set howmuch data will be transferred to memory and from memory in and out ofthe FIFOs, which allows the entire data transfer to effectively runindependent of the line processor. When the transfer count isexhausted/reached, the pipeline hardware can flush the data to memoryand initiate end of transfer checks. The data can be tagged with theadditional error detection and information, i.e. control statusinformation including CRC information can be updated and/or appended tothe data progressing through the pipeline. This effects an additionallevel of error detection by ensuring at the assembly/disassembly part ofthe transfer that all of the expected data was transferred into or outof the FIFO (in agreement with a count in the Middle Machine describedhereinafter). Other mechanisms in the pipe, e.g. the Middle Machine, canthen be most expeditiously informed whether a complete transfer waseffected.

The assembler/disassembler according to the invention is configured toassemble or accumulate burst data, as opposed to moving one piece ofdata at a time, for transfer on the data bus en route to the GM. Thuswhen a threshold quantity of data is accumulated for an efficienttransfer (on a write to GM), the assembler/disassembler passes the burstof data along the pipeline. Similarly, a burst transfer is effected on aread from GM, when a threshold level of space is available in theassembler/disassembler for receipt of data for an efficient transfer.

A buffer dual port ram (DPR) is configured to receive data for bufferingread operations from and write operations to the GM. Data is transferredbetween the assembler/disassembler and the buffer DPR efficiently with amaximum burst size of sixteen, 64 bit memory words. The DPR is used tobuffer and speed-isolate the different transfer rates of the backplaneto/from GM and various data paths on the ESCON interface circuit board.Data transfers between the assembler/disassembler and the buffer DPRpass through Error Detection And Correction circuitry (EDAC)implementing a modified hamming algorithm as known in the art.

According to the invention, a plurality of state machines arranged intothree functional units, an Upper Machine, Middle Machine and a LowerMachine facilitate movement of user data between the buffer DPR and theGlobal Memory (GM).

The Middle Machine controls all data movement to and from the GM.Although not directly in the data path, it is responsible forcoordinating control between the various elements that comprise the datatransfer channels. The Middle Machine is interconnected to the UpperMachine and Lower Machine, and to various global memory and lineprocessor interface, address and control elements.

The Middle Machine provides control and coordination between the Upperand Lower sides of the buffer DPR. The Lower Machine connects to theassembler/disassembler mechanism of each pipe and controls the EDAC. TheUpper Machine connects to the backplane, which in turn connects toGlobal Memory. The actual data transfers between the buffer DPR and GMare controlled by the Upper Machine, and transfers between the bufferDPR and the assembler/disassembler are controlled by the Lower Machine.

On the Lower side, the Lower Machine handles the handshake protocolbetween the assembler/disassembler, the EDAC and the DPR. When anassembler/disassembler needs to transfer data, it sends a request to theMiddle Machine. The Middle Machine arbitrates among the plurality ofLower side requesters, and begins a Lower grant cycle. The MiddleMachine provides the number of the pipe that won the arbitration (the“won pipe number”), along with a Dual Port RAM pointer to the LowerMachine. When the Lower Machine accepts this data, it sends anacknowledge back to the assembler/disassembler which won thearbitration, and begins a data transfer between theassembler/disassembler and the DPR, through the EDAC. Upon completion ofthe transfer, status and ending pointer information are returned to theMiddle Machine.

Similarly, the Upper Machine implements the backplane protocol toaccomplish data transfers between the Global Memory connected to thebackplane and the buffer DPR on the ESCON interface according to theinvention. The Middle Machine determines a pipe that needs servicing.When the Middle Machine finds a candidate it provides the won pipenumber along with a DPR pointer to the Upper Machine. The Upper Machinerequests backplane access and provides the address, command and transferlength to the backplane. When the data transfer phase begins, the UpperMachine manages the DPR address and control signals. Upon completion ofthe transfer, status and ending pointer information are returned to theMiddle Machine Status Register.

The Middle Machine maintains a transfer counter, which is used to pace atransfer and to calculate appropriate backplane burst sizes. The MiddleMachine is further responsible for error management. When an erroroccurs during a transfer (either Upper or Lower), the Middle Machinewill collect status from the Upper and Lower machines, appropriatelypost the error condition in the Status Register, and disable the failingtransfer.

The architecture according to the invention further incorporatessubstantial shared resources including a service processor thatprovides, among other things, configuration and maintenance services.The service processor facilitates messaging between line processors andprovides a single point of contact for a user interfacing with the lineprocessor(s). There is also shared EPROM or FLASH memory from which theservice processor and line processors get their initialization or bootprogram. Shared non-volatile static RAM (SRAM) is provided to store theESCON interface board configuration data and trace logs. A shared timeof day chip records the actual time and date of the real world that isshared between all the processors.

The shared resources include shared Dynamic RAM (DRAM) and Static RAM(SRAM) (“shared memory”). The DRAM can be used to store code, such asdiagnostic code, that is not performance critical. When an error isreceived in a pipeline diagnostics may be run by the processor in thatline. Since the diagnostic code is the same to all processors on theboard there is no reason to keep multiple copies of it, so it is kept inthe shared memory. The shared memory may also be used to store portionsof code that is too large to store locally with the respective lineprocessor.

The shared memory SRAM is further used to provide the ability for theline processors and service processor to communicate via a messagingsystem. The shared memory includes mailboxes that are used tocommunicate between the line processors and the service processor. Theservice processor has the capability of issuing a system managementinterrupt to any or all of the line processors. This interrupt is anindication to the line processor(s) that it should go out to the sharedmemory and read its respective mailbox. In operation, the serviceprocessor can deliver a message, i.e. command, to a line processor'smailbox, for example to tell a line processor to go off-line or on-line.The service processor will write the command into the mailbox and thenassert the system management interrupt on the appropriate line processorthat it wants to read the mailbox. The line processor receiving theinterrupt will take the interrupt vector, read the mailbox, interpretthe command and deliver the appropriate response to the mailbox. Eachline processor has a system management interrupt line going back to theservice processor. Once the line processor has put its response or itsrequest into a mailbox, it will then assert its system managementinterrupt line to the service processor. The service processor iscentral to the interrupt/messaging scheme in that it can interrupt anyline processor and any line processor can interrupt the serviceprocessor.

The shared resources including the service processor are used atstart-up to boot the ESCON interface, and ultimately control thefunctional of the line processors. The service processor also providessystem wide services. When there is a system wide interrupt it isdelivered to the service processor. The service processor decides whatto do with it. It decides whether it needs to interrupt the datatransfer that is in progress. While line processors are responsible fordoing data transfer tasks, they can consolidate certain tasks onto theservice processor to increase system performance. That is, the serviceprocessor can consolidate some of the operations from each of the lineinterfaces to reduce the number of accesses to the backplane thus savingbackplane bandwidth. This is accomplished through the messaging systembetween the processors that is implemented in shared memory.

Features of the invention include provision of an ESCON I/O interfacethat effects high-speed, high efficiency transfer of data between anESCON I/O host and Global Memory (GM) in an integrated cache disk arraysystem. Separation of control information and user data early in thepipeline transfer facilitates highly efficient processing and simplifiedmanipulation of user data traversing the pipeline(s). Segregation ofuser data into a FIFO avoids complex pointer management. Higher levelsof data integrity are ensured by enhanced error detection implemented inthe initial stages of the data traversing the pipes. Data order checkingprovides further enhancement of error detection ensuring the integrityof transferred data.

Further features include increased pipelining and data transfer ratesfacilitated by the assembler/disassembler mechanism which assembles datain order to burst data to the buffer DPR. Bursts of data are moreefficient and yield higher performance than doing single beat datatransfers. The implementation of upper, lower and middle machines tocoordinate data transfers modularizes data management and facilitatespipelining and parallelism.

Avoidance of multiple copies of the code for each processor in eachpipeline avoids wasting resources, such as EPROM or FLASH, and conservesboard real estate. The sharing of substantial resources by the lineprocessors keeps hardware requirements to a minimum. High levels ofintegration are achieved in the interface according to the invention asvarious mechanisms are implemented in very large scale applicationspecific integrated circuits (ASICs) and programmable gate arrays. Thusfewer devices are used and higher reliability is attained.

Separation of facilities and code separates tasks for line processorsand the service processor which simplifies source code modifications andmaintenance. Maintenance and error detection and correction handled bythe service processor minimizes affects on data transfers, as lineprocessors handle data transfers maximizing data throughput andbandwidth. Implementation of a processor messaging system which can beused to prioritize transfers based on most desired operations improvesoverall system performance.

BRIEF DESCRIPTION OF THE DRAWING

The foregoing and other features and advantages of the present inventionwill be more fully understood from the following detailed description ofillustrative embodiments, taken in conjunction with the accompanyingdrawing in which:

FIG. 1 is a block diagram of an architecture of an integrated cache diskarray according to the prior art;

FIG. 2 is a block diagram of an ESCON front end Director or interface ina two-level, dual processor implementation, including ESCON pipes fortransferring data between a host and Global Memory according to theprior art;

FIG. 3 is a block diagram of an ESCON interface for interfacing a hostESCON I/O to a backplane connecting to Global Memory in and integratedcache disk array according to the invention;

FIG. 4 a is a block diagram of receiver logic or hardware implemented inan ESCON protocol engine in the ESCON interface of FIG. 3;

FIG. 4 b is a memory allocation map for a receiver frame store or dualport ram (DPR) in the ESCON interface of FIG. 3;

FIG. 4 c is a map of a Device frame header structure as stored in theDPR of FIG. 4 b;

FIG. 4 d is a map of a Link frame header structure as stored in the DPRof FIG. 4 b;

TABLE I is a description of header fields particular to the headerstructures of FIGS. 4 c and 4 d and processed by the ESCON interface ofFIG. 3;

TABLES II-V are diagrams of pipeline processor interface registersimplemented in the receiver logic of FIG. 4 a;

FIG. 5 a is a block diagram of transmitter logic or hardware implementedin the ESCON protocol engine in the ESCON interface of FIG. 3;

FIG. 5 b is a memory allocation map for a transmitter frame store ordual port ram (DPR) in the ESCON interface of FIG. 3;

FIG. 5 c is a link frame header structure as constructed by the ESCONinterface of FIG. 3 for transmitting link frames;

FIG. 5 d is a device frame header structure as constructed by the ESCONinterface of FIG. 3 for transmitting device frames;

TABLES VI-X are diagrams of pipeline processor interface registersimplemented in the transmitter logic of FIG. 5 a;

FIG. 6 is a block diagram of an assembler/disassembler in the ESCONinterface of FIG. 3;

TABLES XI and XII are diagrams of a command register and a statusregister implemented in the assembler/disassembler of FIG. 6;

FIG. 7 is a block diagram of interconnections between Lower, Upper andMiddle machine functional units implemented via data transfer statemachines according to the invention;

FIG. 8 is a block diagram of a middle machine in a plurality of datatransfer state machines according to the invention;

FIG. 9 is a diagram illustrating the relationship of Global Memoryaddresses and upper and lower pointers for tracking data transfersaccording to the invention;

FIG. 10 is a diagram illustrating upper and lower pointer progression indata transfers according to the invention;

TABLES XIII-XVI are primary address pointer, secondary address pointer,command and transfer length, and status registers, respectively,comprising a register file in the middle machine state machine of FIG.8;

FIG. 11 is a state diagram of a lower arbiter state machine of themiddle machine of FIG. 8;

FIG. 12 is a state diagram of a lower control state machine of themiddle machine of FIG. 8;

FIG. 13 is a state diagram of a lower completer state machine of themiddle machine of FIG. 8;

FIG. 14 is a state diagram of an upper arbiter state machine of themiddle machine of FIG. 8;

FIG. 15 is a state diagram of an upper control state machine of themiddle machine of FIG. 8;

FIG. 16 is a state diagram of an upper completer state machine of themiddle machine of FIG. 8; and

FIG. 17 is a block diagram of a messaging configuration using a serviceprocessor as a central point of contact for a plurality of lineprocessors according to the invention.

DETAILED DESCRIPTION

The mechanisms according to the present invention are implemented in anintegrated cache data storage system involved primarily with moving databetween a host computer and at least one array of storage devices, suchas a disk drive array. More specifically, the mechanisms are implementedas functional elements in a front end ESCON interface, or controller,which moves data between an ESCON I/O interface in the host, as known inthe art, and a Global Memory or cache integrated in the storage system.

ESCON Interface Overview

The ESCON architecture interface protocol as known in the art generallycomprises three levels of communication between devices connected overthe ESCON link or interconnection. At a physical level, “sequences” arepassed between devices over the ESCON link. ESCON sequences are bitsthat are specially formatted and/or combined to have special meaningsrelating to the interconnection or physical interface. Link level framesor “link frames” are a second level of ESCON communication that furtherdeal with or define the physical interconnection of the ESCON interfaceprotocol at a higher level than the sequences. Basically, link framesare used to maintain the virtual connection between ESCON clients, i.e.the devices interconnected over ESCON. A third level, the highest level,of ESCON communication is effected via “device frames.” Device framesare used to pass control and status information between ESCON clients,and they are used to facilitate the functionality derived from theinterconnection between ESCON clients.

In this illustrative embodiment describing an ICDA interconnected to ahost computer over ESCON, device frames are the primary mechanism forpassing “user data” or customer data that is being passed between thecache and the host for storage in the ICDA. Device frames or link framescan be used to pass control information in the form of commands andparameters controlling the operation of the ESCON interface according tothe invention and/or the ICDA. Each frame or packet of ESCONinformation, generally, contains a header identifying the frame andframe type, a trailer and a data information block (DIB or user data).The trailer portion of the frame is usually Cyclical Redundancy Check(CRC) information. CRC information, as known in the art, is used inchecking for errors in data transmission. In this illustrativeembodiment, ESCON device frames are user data frames having informationin the DIB portion that is to be stored in or has been retrieved fromdisk. The user data (DIB) portion of the ESCON standard link controlframe typically are parameters to a command argument. Parameters issuedas part of an ESCON frame include an indication that the data isinformation about the transfer that is not part of the transfer.

An ESCON interface circuit board illustrated in FIG. 3, generally, is aneight pipeline interface between the host and Global Memory (“GM”). Eachof the eight pipelines or pipes moving data is substantially identicalin terms of structure and functionality. In the illustrative embodimentof the invention in FIG. 3, the ESCON interface according to theinvention is implemented as a circuit board that is connected to anadapter board 20 that is physically connected to the host ESCON I/Ofiber optic cabling (not shown) for ESCON communication. The adapterboard 20 comprises a plurality of fiber optic transceivers 22 (FOT) thatreceive information from the host formatted according to the ESCONprotocol known in the art.

Each FOT on the adapter board 20 in this illustrative embodiment is aconventional Part No. 1401BB made by Seimens. The FOTs 22 on the adapterboard 20 pass ESCON standard serial information going to the ESCONinterface board to a respective HOTLINK receiver 24 (Cypress Part No.CY78933) in the pipeline. The HOTLINK receivers 24 convert the ESCONstandard serial data into ESCON standard parallel data. The receiver 24uses 8b10b encoding to take 10 serial bits from the fiber optic channeland convert that to 8 bits parallel plus parity. The receiver 24 thenadds a special character to the data as a flag to indicate that there isactually 8 bits of data and to indicate where a data frame starts andwhere the frame ends.

Similarly, HOTLINK transmitters 26 (Cypress Part No. CY78923) in thepipeline receive ESCON standard parallel information coming from theESCON interface board for transmission to the FOTs 22 on the adapterboard 20. The HOTLINK transmitters 26 convert the ESCON standardparallel data into ESCON standard 10 bit serial data for receipt by theFOTs 22 and transmission to the fiber optic channel. The HOTLINKreceivers and transmitters are described in detail in the Cypress DataBook, which is incorporated herein by reference.

In a pipeline, a HOTLINK receiver 24 and a transmitter 26 are paired andelectrically interconnected to an ESCON protocol engine 28 implementedin a Field Programmable Gate Array (FPGA) (Part No. XC4013E-3). ParallelESCON standard data is transferred between the protocol engine 28 andthe HOTLINK receiver 24 and transmitter 26. The ESCON protocol engine iscomprised of receiver logic that interfaces with the HOTLINKS receiver24 and ESCON transmit logic that interfaces to the HOTLINKS transmitter26. The protocol engine in a pipe processes the information, e.g.frames, sent from/to the ESCON host I/O, as described in detailhereinafter.

The protocol engine 28 ESCON receive logic is electricallyinterconnected to a receive frame dual port ram (DPR) 30 providing aframe header store. The receive frame DPR 30 stores headers and otherinformation, such as commands and parameters, which is not customer datato be stored on disk. If the frame is something other than a user dataframe, such as a command frame or a link control frame, then theprotocol engine sends the frame information to the receive frame DPR 30.Because the receive frame DPR 30 is a dual port ram the protocol engine28 can be storing data over one port while a line processor is readingit out of the other port.

The protocol engine 28 ESCON transmit logic is electricallyinterconnected to a transmit frame DPR 32. ESCON frame information beingsent from the ESCON interface according to the invention is assembled inthe transmit frame DPR. Ultimately, an assembled ESCON frame, includingany required user data, is transmitted through the protocol engine andto the adapter board 20 for transmission to the host ESCON I/O.

In addition to being connected to the receive frame DPR 30, the protocolengine 28 ESCON receive logic is electrically connected to a set ofreceive First-In-First-Out registers (“receive FIFO”) 34. The protocolengine 28 discriminates between frame/header information and user data,and stores the user data in the receive FIFO 34, as data progressesthrough the pipeline(s) toward GM (Global Memory) as described in detailhereinafter. Similarly, the protocol engine transmit logic iselectrically connected to a set of transmit FIFOs 36. User data to beconstructed into ESCON frames for transmission back to the host ESCONI/O is stored temporarily in the transmit FIFO 36.

An assembler/disassembler 38 according to the invention is electricallyinterconnected to the receive FIFO 34 and to the transmit FIFO 36. Theassembler/disassembler 38 is in turn electrically interconnected to adata bus (“the SDIO bus”) 40 over which data is transferred goingto/from the backplane and GM. On a write to GM theassembler/disassembler 38 receives user data from the receive FIFO 34and assembles the data into memory words and/or blocks of memory wordsfor bursting onto the SDIO bus 40. On a read from GM theassembler/disassembler 38 receives memory data from the SDIO bus 40 anddisassembles it into bytes for temporary storage in the transmit FIFO36.

The SDIO bus 40 facilitates communication of data transferred between abuffer dual port ram (DPR) 42 and the assembler/disassembler 38. Thebuffer DPR 42 is configured to receive data for buffering readoperations from and write operations to the GM. Data transfers betweenthe assembler/disassembler 38 and the buffer DPR 42 pass through ErrorDetection And Correction circuitry (EDAC) 44.

An arrangement of state machines, three in this illustrative embodimentcomprising a Lower Machine 46, a Middle Machine 48 and an Upper Machine50, facilitate movement of user data between the buffer DPR 42 and thebackplane to Global Memory (GM).

The Lower Machine (“LM”) 46 controls “lower side” data transfers,including the handshake protocol between the assembler/disassembler 38,the EDAC 44 and the buffer DPR 42, for transferring data on GM reads andwrites. For example, when an assembler/disassembler 38 needs to transferdata to GM (i.e. a write operation), it sends out a request, which issubject to arbitration as described in detail hereinafter. The LM 46receives information regarding a pipeline data transfer and sends anacknowledge back to the assembler/disassembler which won thearbitration. The assembler/disassembler that owns the right to transferbegins a data transfer over the SDIO bus 40, through the EDAC 44 to thebuffer DPR 42. Similarly, on read operations the LM 46 controlstransfers from the buffer DPR 42 through the EDAC 44 over the SDIO bus40 to the assembler/disassembler(s) 38.

The Middle Machine 48 (“MM”) effectively oversees control of all datamovement to and from the GM. Although not directly in the data path, itis responsible for coordinating control between the various elementsthat comprise the data transfer channels. The MM 48 is interconnected tothe LM 46 and the Upper Machine 50, and various global memory data pathelements. The MM 48 provides control and coordination between the upperand lower sides of the buffer DPR 42. The lower (or Lower Machine) sideof the MM 48 connects to the assembler/disassembler 38 of each pipe andthe LM 46. The upper (or Upper Machine) side of the MM 48 connects tothe backplane and the Upper Machine, which in turn connects to GM. Theactual data transfers between the buffer DPR and GM are controlled bythe Upper Machine and Lower Machine.

The Upper Machine 50 (“UM”) implements the backplane protocol toaccomplish data transfers between the GM connected to the backplane andthe ESCON interface according to the invention. The MM 48 determines apipe that needs servicing. The UM 50 receives, from the MM 48,information regarding the pipe to be serviced and requests backplaneaccess. The MM 48 provides the address and the UM 50 provides thecommand and transfer length to the backplane over a plurality oftransceivers 52, 54, 56. When the data transfer phase begins, the UM 50manages the buffer DPR 42 address and control signals, and provides aclock on write operations. Data is transferred from the buffer DPR 42 tothe backplane over a backplane transceiver 58.

Referring still to FIG. 3, The pipeline elements generally describedhereinbefore, and pipeline transfers described in greater detailhereinafter, are controlled in each pipeline by a respective pipelineprocessor 60. That is, commands from the host ESCON I/O for transferringdata or otherwise interfacing with GM through the ESCON interfaceaccording to the invention are received and executed by the pipelineprocessor. In this illustrative embodiment the pipeline processors arepower PC 603E or the like. Each pipeline processor 60 has localresources including a second level cache 62 which is used for operatingsoftware, local variables, and control information. Each line processor60 has a dynamic RAM store 64 for various processor storage tasks, suchas configuration information and trace logs. A transceiver ASIC 66implements a bus gasket function that effectively provides a pluralityof selectable transceivers for interconnections between the lineprocessor 60 and the various busses to which it has access. A processorcontroller 68, in the form of a MACH5LV-384/192, implements controllogic for controlling a respective power PC line processor 60 andtransceiver ASIC 66 according to the functionality describedhereinafter.

The architecture according to the invention further incorporatessubstantial shared resources, configured as illustrated in FIG. 3,including a service processor 70 that provides configuration,maintenance and operation services. The service processor 70 facilitatesmessaging between line processors and provides a single point of contactfor power-up and for a user interfacing with the ESCON interface(s). Theservice processor 70 has the same local resources as the lineprocessors, including a second level cache 72, a dynamic ram store 74, atransceiver ASIC 76 and a processor controller 78. The service processor70 communicates with GM via a Memory Data Register device (“MDR”) 80,rather than through an associated assembler/disassembler because theservice processor has no need to assemble/disassemble bursts of data fortransmission to/from GM. It should be noted that the MDR issubstantially similar to the assembler/disassembler 38 describedhereinafter, except that it does not include a FIFO mechanism forassembling bursts of memory data words. The line processors 60 also havean MDR, implemented in the assembler/disassembler 38, used to transfercache management and control information to/from the GM.

The service processor 70 provides a single point of contact, through aSmall Computer System Interface (SCSI) ASIC 71, for monitoring operatingparameters in the ICDA. The SCSI ASIC 71 includes an RS 232 interface asknown in the art, over which information relating to operatingparameters such as power and temperature can be transmitted andreceived. Similarly, network connection can be effected and monitored bythe service processor acting as a single point of contact forcommunication over Ethernet facilities as known in the art, which areincluded in the SCSI ASIC 71.

Other shared resources available to and shared by the line processorsand service processor, include shared EPROM or FLASH memory 82 fromwhich the service processor and line processors get their initializationor boot program. Shared non-volatile dynamic RAM (NVD) 84 is provided tostore the ESCON interface board configuration data and trace logs,including error messages. A shared time of day chip 86 that is sharedbetween all the processors records the actual time and date of the realworld. The shared resources further include shared, synchronous DynamicRAM (DRAM) (“shared memory”) 88 used to store code, such as diagnosticcode, and device tables and/or other large data that is not performancecritical relating to the circuit board. When an error is received in apipeline, diagnostics may be run by the processor in that line. Sincethe diagnostic code is the same to all processors on the board there isno reason to keep multiple copies of it, so it is kept in the sharedmemory. The shared memory may also be used to store portions of code, ordata, too large to store locally with the respective line processor, andwhich is not performance critical. The shared memory is further used toprovide the ability for the line processors and service processor tocommunicate via a messaging system, described in detail hereinafter. Theshared resources are accessible over various busses, including a sharedbus (cntl, data and addr) via transceivers 90, illustrativelyimplemented using Philips 899 transceiver logic. Access to the sharedresources described hereinbefore is provided to the line processors 60and service processor 70 as a result of arbitration effected by a sharedcontroller CPLD 91. In addition to an arbitration mechanism thatdetermines which line processor (or the service processor) gets accessto the shared resources, the shared controller CPLD 91 effects controlof timing of access to the shared resources, and includes a DRAMcontroller for the SDRAM 88.

The ESCON Protocol Engine

Referring now to FIGS. 3 and 4 a, the ESCON interface according to theinvention includes an onboard ESCON protocol conversion device orprotocol engine 28 in each pipeline. Each protocol engine 28 comprisesreceiver logic 100 (FIG. 4 a) and transmitter logic 200 (FIG. 5 a).Generally, the protocol engine 28 is configured with the capability ofdistinguishing data which is customer, or user, data that is to bestored on a disk or is read from disk versus the headers and informationthat defines control frames in which the data is presented. The receiverlogic 100 is electrically connected for passing data to the receiveframe DPR or Frame Store 30 and the receiver FIFO 34.

The receiver logic or frame reception logic 100 comprises an inputregister 102 that receives eight bit wide data from the HOTLINK receiver24. The input register 102 is a register located in the I/O section (an“IOB”) of the FPGA device implementing the protocol engine (rather thanone of the internal registers or “CLBs”). A second input register 104 isused to receive HOTLINK control information from the receiver 24.HOTLINK control information is received by Loss of Synch detection logic(LoS DETECT) 105. The LoS detect logic checks to see if there has been aloss of sync condition on the ESCON link. Loss of synch is checked foradherence to the ESCON protocol standard, in terms of the loss of syncdefinition.

At the input to the receiver logic 100 an input pipeline is implementedhaving five stages, the first being the input register 102. As each byteof information is received it is initially buffered in the inputregister 102 as a first stage of the input pipeline. Four internalregisters 106, 108, 110, 112, comprise the remaining four registers inthe input pipeline. The 8 bit wide data path defined by the inputpipeline facilitates operations on fields of the input data before anyuser data is buffered downstream in the receive FIFO 34 or the receiveframe store DPR 30.

As a data byte goes from the first pipeline buffer 102 to the secondpipeline buffer 106 it is available to frame decode circuitry 114. Theframe decode circuitry 114 is logic in the protocol engine FPGA thatchecks that the structure of the frame is correct in terms of ESCONprotocol. For instance, starting and ending delimiters are checked whichdesignate the start and end of a frame, and headers and trailers of thepackets or frame that are coming in are checked to determine that theyare the proper size. The frame decode logic 114 is configured as afunction of the requirements of the ESCON protocol.

Sequence detector logic 116 in the protocol engine FPGA also receiveseach data byte as it goes from the first pipeline buffer 102 to thesecond pipeline buffer 106. The sequence detector 116 checks for thespecial ESCON protocol defined sequences that have special meanings inthe protocol. ESCON sequences are not part of normal frames. Thesequence detector 116 facilitates generation of interrupts to the lineprocessor as a function of changes in state indicated by ESCON sequencesas described in detail hereinafter.

Data bytes transferred through the input pipeline stages 102, 106, 108,110, 112 are processed by an ESCON CRC checker 118. The ESCON CRCchecker checks data integrity according to the CRC algorithm that isdefined in the ESCON protocol specification. The ESCON CRC checker 118issues an ESCON CRC error if the CRC for the ESCON frame is erroneous. Alocal CRC generator 120 accumulates a CRC on the data, i.e.non-header/frame information, as it goes out to the receive FIFO 34. Thelocal CRC generator provides an exclusive-OR operation of the datafield(s) during transfer. Thus the ESCON CRC checker 118 checks theintegrity of the entire frame, headers, trailers, and informationtherebetween. The local CRC generator 120 turns on when the data portionof this field is beginning. The accumulated local CRC is calculated fromthe data that is stored in the receive FIFO 34, and stored with theframe header in the receiver DPR 30.

A receiver machine or state machine 122, along with providing otherfunctionality and timing relationships as described herein, controlswhen the ESCON CRC and the local CRC mechanisms turn on and off, basedon frame structure. The receiver machine is started when a new frame isbeing detected in the pipeline stages as coming in from the HOTLINKinterface 24. The machine is idle when there is nothing going on, i.e.it is receiving idle character sequences. When the frame decode logic114 indicates a delimiter that is a start of frame, the receiver statemachine triggers and then sequences through all the incoming framefields (the header fields for device frames and link frames areillustrated in FIGS. 4 b-4 d). The receiver state machine is controllingthe building of particular fields that will be stored with the header,such as the local CRC (SCRC) and parsing of other fields, such as theinformation field identifier (IFI) which identifies whether a deviceframe is a data frame or a control frame.

The receiver machine provides control signals for running the mechanismsfor building the header fields. Along with building the SCRC into theheader, the receiver machine 122 starts a DIB counter 126 which countsbytes of the data information block in order to generate a DIB lengthfield (DLEN) which is provided to the frame MUX 124. The DIB length isbuilt into the header information stored in the receiver DPR so thatpipeline control software (running on the pipeline processor) can accessthe DPR and know how big the user data portion of the frame is so it canthen instruct the assembler/dissembler (FIG. 3, 38, described in detailhereinafter), to move data accordingly based on its size.

A Frame Status Generator 128 provides a status register that accumulatesan Fstatus field to append on the frame header for storage in the DPR.The status field indicates whether any errors occurred during receptionof the frame. The Fstatus field provides status per frame in thedual-port ram.

The status field (status), local CRC (SCRC), data length (DLEN) and DIBdata are provided as inputs to a frame MUX 124. The frame MUX 124 alsoreceives as an input a DIB pointer (SV_DP) from a DIB pointer register130. The frame MUX 124 is controlled as well by the receiver machine122. The frame MUX 124 is sequenced to provide the header fields forstorage. The receiver machine issues control signals to appropriatelyposition information in the data path to determine how the data isformatted inside the receive FIFO 34 and inside the receive frame DPR30.

The receiver state machine 122 also controls addressing of the receiveFIFO 34 and the receive frame DPR 30. As the headers are stored in thereceive frame DPR 30 the location in the FIFO or DPR is added to theheader as a Data Address (Data Adr) and DSLT field via a header pointerregister 132 and a DIB pointer register 134. Thus the status field(status), local CRC (SCRC), data length (DLEN), DSLT and Data Adr fieldsare determined and available. The destination link address and otherheader fields are available coming off of the link. Accordingly suchinformation can be appended and the frame header stored in the receiverframe store or DPR 30. The receiver state machine includes DPR controllogic 136 which controls the chip enable for the DPR.

The dual-port ram 30 is divided up into slots and basically one slot isdedicated to one frame. When a frame is received it is provided with itsset of parameters or fields in the DPR 30. With the header informationconveniently available in the frame store DPR 30, the line processor canread out all the information for a slot/frame and act accordingly. Theline processor can subsequently move to the next slot in the dual-portram and get the next receive frame and parameters. Frame status can bechecked on a per frame basis, and the software can implement commandssuch as retry or resend the information. Likewise, the software can givethe hardware instructions to move the data that is in the FIFO up thepipeline, through the assembler/disassembler 38 and eventually into theglobal memory.

The receiver state machine 122 includes FIFO control logic 138. When itis determined that a write (i.e. toward GM) is necessary and DIBinformation is to be written to the FIFO, the receiver machine controlsthe write enable for the FIFO 34. There is an FF (FIFO Full) flag on theFIFO that comes back into the FIFO control 138. The FF signal indicatesthat the FIFO 34 is full. If the FIFO is full and a write is attempted,an error condition results in the form of a FIFO overflow.

The linear feedback shift register 140 in conjunction with the paritygenerator 142 in the data path implement the enhanced parity mechanismas described hereinafter. Substantially identical circuitry isimplemented in the assembler/disassembler 38 to check addressingintegrity across the FIFO. The enhanced parity circuitry in the ESCONprotocol device is configured to run in synch with the enhanced paritycircuitry in the assembler/disassembler 38.

The line processor interface to the receiver logic 144 is implemented ina plurality of registers contained in the ESCON protocol engine FPGA.The control and status registers include an ESCON receiver controlregister, an ESCON receiver status register, ESCON receiver maskregister and an ESCON receiver diagnostic register. These control andstatus registers are illustrated in TABLES II-V, respectively. The lineprocessor 60 has access to these registers through the transceiver ASIC66 in order to control and check the status of the ESCON protocol enginereceiver logic.

The ESCON interface protocol engine frame reception logic receives ESCONframes transferred by the host ESCON I/O according to the ESCON format.The frame reception logic processes the ESCON frames via the receiverlogic frame control circuitry including the receiver machine. Thereceiver logic can be in one of several states, including: a Stoppedstate; a No Frames Received state; a Receive Frame state; a ReceiverFull state; or a Receiver Overflow state. The Stopped state is enteredupon power-up or Reset. The receiver also stops when an overflowcondition is detected or when a frame being received is terminated dueto a severe error condition. When no frames are being received by thereceiver logic of the protocol engine, and it is not in the Stopped orother state, it is in the No Frames Received (or “idle”) state. In theReceive Frame state one or more frames are being or have been receivedand the number of frames received is less than a limit specified by aparameter RcvCtlBINDX. In the Receiver Full state the Frame Store bufferis full according to the limit specified by RcvCtlBINDX. The receiverhardware will not store anymore received frames while in the ReceiverFull state. The receiver logic of the protocol engine 28 goes into theReceiver Overflow state when a frame is received while the Frame Storebuffer is full. After a Receiver Overflow the receiver hardware goesinto the Stopped state, and needs to be explicitly re-started bysoftware via the RcvCtl register.

Various error conditions that may occur during receipt of frames aredetected by the protocol engine receiver logic. The error conditionsthat the receiver logic is configured to detect can be divided into twocategories: errors that cause the current frame reception to stop anddisable further frame reception; and errors that only cause the currentframe reception to stop.

Errors that cause the current frame reception to stop and disablefurther frame reception, require that the protocol engine receiver logicbe re-enabled under software control. Such errors include: framereception ended by detection of a new sequence; frame reception ended bydetection of a Start of Frame (SOF) delimiter; detection of a loss ofsynchronization between the host ESCON I/O and the ESCON interfacereceiver logic; detection of a Frame Store Overflow; and detection of aFIFO Overflow.

During frame reception, if the receiver logic detects an error conditionof the second type, i.e. errors that only cause the current framereception to stop, the error is reported in the frame status field ofthe frame header. Errors that only cause the current frame reception tostop include: ESCON-CRC errors; a frame reception ended by detection ofa Control character; a frame reception ended by detection of an Invalidcharacter (violation); incoming frame size exceeding 1035 bytes; framereception ended by detection of two consecutive Idle characters; framereception ended by detection of an ABORT delimiter; and frame receptionended by detection of an end-of-frame (EOF) delimiter.

Even though frame reception is stopped if the foregoing error conditionsare detected by the protocol engine receiver logic, ESCON sequencereception continues. That is, the protocol engine receiver logic 100continues to monitor the receiver channel for a data stream from thehost ESCON I/O. The receiver logic 100 continuously monitors thereceived data stream for the following ESCON I/O sequences:

-   -   IDLE 16 or more consecutive K28.5 chars    -   UD Unconditional Disconnect    -   UDR Unconditional Disconnect Response    -   NOP Not Operational    -   OFL Offline    -   RSVD Reserved

The receiver logic 100 is configured to receive and check an ESCON CRC,or cyclic redundancy check information, provided by the host ESCON I/Owith its data transmission. The ESCON-CRC is not stored by the receiverlogic. It is checked for correctness and flagged in a Frame Statusregister (Fstatus) if it is not correct.

Synchronization between the host ESCON I/O and the ESCON interfacereceiver logic is required in order to avoid an error which will causethe current frame reception to stop and disable further frame reception.Accordingly, the receiver hardware always monitors the received datastream to update its synchronization state according to the ESCONspecification. A Loss of Sync is detected if 4 or more code-violationsare received without any intervening sequence of 16 consecutive Idlecharacters. Synchronization is regained when 16 or more Idle charactersare received.

Bit Error Rate (BER) monitoring is done by software in the illustrativeimplementation in order to measure line degradation. A bit (BER bit) isset by Receiver hardware whenever a code-violation is detected in thereceived data stream and this bit is monitored and cleared by software.

The ESCON Processor in the receiver logic 100 of the ESCON protocolengine 28 receives data streams from the host ESCON I/O and effectsdiscrimination between user data (DIB) in a frame and frame/headerinformation. The protocol engine 28 receiver logic 100 is connected tothe receive frame DPR (“Frame Store”) 30, and the protocol engine 28ESCON receiver logic 100 is electrically connected to a set of receiveFirst-In-First-Out registers (“receive FIFO”) 34.

As illustrated in FIG. 4 b, the receiver Frame Store is an 8K-bytebuffer divided into two sections of fixed size. A first portioncomprising 1K-bytes is allocated for storage of all frame headers. Asecond portion comprising the remaining 7K-bytes is reserved for storageof frame data. The 1K-byte frame header section is divided into 32 slotsof 32 bytes each. The 7K byte frame data section is divided into 7 slotsof 1k-byte each. The frame header section of received Frame Store 30 isoperated upon as a circular queue of 32 slots, meaning that up to 32consecutive frames can be received and buffered before frames startgetting discarded.

The frame header section circular queue is managed by two pointers: aWrite Index (WINDX); and a boundary index (BINDX). WINDX represents theheader slot to which the NEXT received frame will be written byhardware. WINDX is updated on any new frame reception by the ESCONreceiver logic. Before reading WINDX, it must be updated to ensure thatthe most current value is read back. BINDX represents the last slot thatcan be used to receive frames. BINDX is set by software to the slotnumber of the last frame that is allowed to be received by the receiverhardware. The receiver hardware checks the WINDX pointer against theBINDX pointer. These two pointers are aligned to 32 byte boundaries.Each frame header received from the link is put into a separate headerslot indicated by WINDX. In addition to the actual frame header receivedfrom the link, the receiver hardware also stores some extra informationpertaining to the status of the received frame. Once the ESCON protocolengine receiver hardware detects that WINDX=BINDX+1 it will discardfurther incoming frames until BINDX is updated by the software.

User data or DIB information (i.e. to ultimately be written to disk) isstored either in the Frame Store buffer 30 or in the receive FIFO 34,depending on the frame type and instruction from the software. The DIBportion of Link-Control frames are always stored in the Frame Storebuffer with the header. The header structure of a link control frame isillustrated in FIG. 4 d. The DIB portion of Device frames is stored inthe Frame Store buffer if a receiver control bit (RcvCtl.G) is equal tozero or in the receive FIFO if that receiver control bit (RcvCtl.G) isequal to one. The header structure of a device frame is illustrated inFIG. 4 c. The definition of bits in the frame headers is set forth inTable I. If the device frame DIB is stored in receive FIFO, the DSLTfield, the three lower order bits in the data address (Data Adr) in theframe header will be 0x0. If the Device frame DIB is stored in the FrameStore the DSLT field will 0x1 through 0x7 depending on which of the 7 1k-byte slots the hardware put the frame into. All data written to theReceive Frame Store will have ODD parity.

The user data is effectively concatenated by stripping the frame/headerinformation and storing the separated user data in the FIFOs. The userdata is subjected to enhanced error detection as it moves through thepipeline(s) according to the invention. The protocol engine isconfigured to compute Cyclic Redundancy Check (CRC) error detectioninformation on the separated (concatenated) user data after strippingthe frame/header information. Thus although the CRC data transferredfrom the host with the data frame is stripped, i.e. not stored, CRCinformation is computed by the protocol engine for the separated userdata at a point close to the physical interface delivering the data tothe ESCON interface according to the invention. The integrity of thedata can therefore be tested again as the data moves through thepipeline(s), i.e. from the FIFO to the assembler/disassembler 38.

Synchronization between the receive FIFO 34 and theassembler/disassembler 38 is ensured in the implementation according tothe invention by checking the integrity of the simplified pointermanagement implemented upon entering the user data in the FIFOs.Effectively no pointer management is required, given the nature of theFIFO and that the first data entering the FIFO is the first dataavailable from the FIFO. The order, and thus integrity, of data movingthrough the FIFO is checked by an enhanced parity mechanism. The userdata is stored as bytes in the FIFO which is nine bits wide tofacilitate a parity bit, as known in the art. This parity checkmechanism is enhanced by a pseudo-random number generated in the linearfeedback shift register in the protocol engine FPGA. The pseudo-randomnumber is used to determine whether the parity should be odd or even forthat byte when the data bytes enter the assembler/disassembler 38, i.e.exits the receive FIFO 34.

The enhanced parity mechanism effectively checks for any disparitybetween parity in the receiver logic of the protocol engine and theassembler/disassembler (which has substantially identical LFSR/enhancedparity circuitry). Thus longitudinal data synchronization can bemaintained across the FIFO. In this implementation, the receiver logicalternately generates even/odd parity for each byte that is written tothe receive FIFO based on the pseudo-random repeating pattern. Theassembler/disassembler alternately checks the even/odd parity on eachbyte that it reads from the FIFO based on the pattern. The receiverlogic and the assembler/disassembler will continue to run in this modeuntil an error is detected or a reset occurs. This enhanced paritymechanism is disabled after reset and can be enabled by asserting asignal RcvMM.EnDisp. If RcvMM.EnDisp is not set, i.e. RcvMM.EnDisp=0,then all data written to the FIFO will have odd parity.

The enhanced parity LFSR is implemented according to particular rules.The LFSR is initialized to a “seed” value at power-up and upon reset.After reset, the LFSR will be enabled only when bytes are written to orread from the receive FIFO 34. The LFSR implements a polynomial togenerate the pseudo-random number, as known in the art, which is of alength so that the repeating pattern is matched to the depth of theFIFO. The pseudo-random number generated in this illustrative embodimentimplements parity according to a repeating pattern wherein the firstbyte written into the receiver FIFO is even parity, the second byte isodd parity, the third byte is even parity, and the fourth, fifth andsixth bytes are odd parity. The LFSR will run continuously in this modeuntil a reset occurs or the receive FIFO is flushed by theassembler/disassembler. This ensures that the receiver logic in theprotocol engine and the assembler/disassembler hardware are always insync. A substantially similar mechanism is implemented to synchronizethe assembler/disassembler 38 and the transmit FIFO 36 for data bytesbeing transferred therebetween, i.e. on a read operation.

The receiver logic is configured to generate interrupts to therespective line processor in the case of four types of events, definedas: Machine Check Interrupts: Frame Interrupts, Sequence Interrupts, andStopped Interrupts. For all of the interrupts, a bit is provided toenable/disable or mask a specific source of the interrupt.

Machine Check Interrupt occurs when there is a data parity error on awrite to any receiver register, or when there is an address parity erroron a read/write to any receiver logic register. If a data parity erroris detected on a write to any register, a Data Parity Error (DPE)interrupt is generated. Receiver status register bit RcvStat.DPEindicates this condition. The DPE interrupt and RcvStat.DPE are clearedwhen a receiver control signal RcvCtl.CMC is asserted to clear MachineCheck Interrupts. If an address parity error is detected on a read orwrite to any register, an Address Parity Error (APE) interrupt isgenerated. There is no status bit in the receiver status register toindicate this condition. The APE interrupt is cleared when RcvCtl.CMC isasserted. Machine Check Interrupts may be indicative of a majormalfunction in the ESCON interface and are always enabled.

Frame Interrupt occurs anytime a new frame is received by the ESCONprotocol engine receiver logic and the receiver logic is enabled togenerate frame interrupts, by a frame enable interrupt signal FrmEn.Thus Frame Interrupt is generated when a frame is received and FrmEn=1.Frame Interrupt is indicated in the receiver control register byRcvCtlFRI. In addition to the FRI bit in the control register, there isanother bit in the Receiver Status register called RcvStat.FRM whichindependently indicates frame. RcvStat.FRM can be polled even if theinterrupt is disabled.

Sequence Interrupt occurs when various sequences are detected in thedata stream as the receiver logic monitors the data stream for ESCONprotocol sequences. Sequence Interrupt occurs whenever any of the ESCONprotocol IDLE Sequence, RSVD Sequence, Valid Sequences (UD, UDR, OFL,NOP), or Loss of Sync (LoS) are detected. Sequence Interrupt is affectedby enables VsqEn, IdlEn, RsqEn, and LoSEn. It is generated when a Validsequence is detected and VsqEn=1, when Idle Sequence is detected andIdlEn=1, when Rsvd Sequence is detected and RsqEn=1, or when Loss OfSync is detected and LoSEn=1. It should be noted that during framereception, the Idle sequence might be detected between frames. To avoidthese interrupts IdlEn should be off when frame reception begins. Itshould also be noted that Sequence Interrupt will be active due to LoSas long as the LoS condition persists. The LoSEn should be turned OFFafter software detects this condition, until sync is regained (which isindicated by the detection of the Idle sequence).

All of the Sequence Interrupt conditions are indicated by RcvCtl.SQI inthe receiver control register. The current state of Sequence-receptionand Synchronization can be read from the receiver status register(RcvStat) independent of the Interrupt enables. Receiver status registerbits RcvStat.Seq and RcvStat.LoS indicate that the referenced Sequencesand Loss of Synchronization conditions, respectively, have occurred sometime in the past. A receiver status bit RcvStat.SQI, and RcvStat.Seq andRcvStat.LoS are cleared immediately after the RcvStat register has beenread by the ESCON Processor. RcvStat.Seq and RcvStat.LoS are immediatelyupdated with the current state of sequence reception and sync detection.

Stopped Interrupt is generated by the receiver logic if the receiverlogic goes into the Stopped state due to a severe error, such as a FrameStore or FIFO Overflow condition. The receiver logic could also go intoa Stopped state, causing a Stopped Interrupt, when a received frame isterminated by detection of Loss of Synch or a Start Of Frame delimiter,or when the frame is terminated by IDLE characters.

The Stopped Interrupt is enabled by asserting RcvMM.StpEn. It isgenerated when the frame reception logic goes into the STOPPED state andthe enable is set to one (i.e. RcvMM.StpEn=1). As discussedhereinbefore, frame reception stops in the event of: a Frame StoreOverflow; a FIFO Overflow during frame reception; a sequence RSVD duringframe reception; a LoS detected during frame reception; and a SOFdetected during frame reception.

A receiver status register bit, RcvStat.Stp, indicates the STOPPED state(i.e. whether the frame reception is stopped or not) and is independentof the state of RcvMM.StpEn. There is no status bit indicating that aStopped Interrupt was generated. Since the only condition that causesStopped Interrupt without Frame Interrupt is Frame Store Overflow, thesoftware has to check a receiver status register bit RcvStat.FOVF todetect the Stopped condition if the Frame Interrupt status bit is equalto zero (RcvStat.FRI=0). The status bits RcvStat.Stp, RcvStat.FOVF,RcvStat.GOVF and RcvStat.ABN are cleared by asserting a receiver controlsignal RcvCtl.FSt.

The Stopped Interrupt remains active as long as the receiver is in theStopped state. If the frame reception is not restarted when the STOPPEDcondition has been detected, then a signal to deactivate the StoppedInterrupt, RcvMM.DtpEn, should be off until the frame reception isrestarted by the FrmStart command. The RcvMM.StpEn bit can then beasserted.

On Power Up or After Reset command, the receiver logic will be in aninitial state wherein WINDX=BINDX=0, i.e. the header portion of theFrame Store will be empty. Also, Frame Reception will be in the Stoppedstate, and the receiver logic enhanced error detection mechanism will beoff. All interrupts after power-up or reset are disabled, except theMachine-check interrupt. In order to reset the receiver logic, a Resetcommand, RSTO, is issued as an output signal from the line processor.The receiver Reset command RSTO is activated by the RESET instructionexecuted by the line processor.

Referring now to FIGS. 3 and 5 a, each protocol engine 28 comprisestransmitter logic 200 (illustrated in the block diagram of FIG. 5 a)that is configured to receive information from the transmit FIFO 36 andthe transmit frame DPR 32. The ESCON interface protocol engine frametransmission logic assembles ESCON frames for transfer to the host ESCONI/O according to the ESCON format. The transmit logic 200 comprises atransmit data path that effectively represents a convergence point forthe transmit FIFO 36 and the transmit frame DPR 32. The transmit frameDPR is a source for header information and the transmit FIFO 36 is thesource for user data coming from the GM and being transmitted to thehost ESCON I/O over the ESCON link. A transmit state machine 202controls the timing relationships and functionality, described in detailherein, required to build frames for transmission over the link inaccordance with the ESCON protocol.

A nine bit data path from the transmit FIFO 36 and a nine bit data portfrom the transmit DPR 32 are provided as inputs to a transmit frameassembly MUX 204. The transmit state machine 202 implements transmitframe DPR control logic 206 and transmit FIFO control logic 208 which,among other things, control the transmit frame MUX 204 to selectivelyprovide header information (and/or DIB information) from the transmitframe DPR 32 and DIB or user data information from the transmit frameFIFO 36 for convergence into frames on the transmit data path. The DPRcontrol logic 206 also provides the chip enable signal(s) to thetransmit frame DPR 32 for accessing the information therein. Thetransmit FIFO control logic 208 provides the enable signal(s) for thetransmit FIFO 36, and receives an error signal (EF) indicative of atransmit frame FIFO error.

Basically, under control of the transmit state machine 202 the transmitFIFO is accessed to obtain user data for building device frames totransmit customer data to the host ESCON I/O. The transmit frame DPR isselectively accessed to obtain header information built in the DPR forconstructing control and data device frames, or command and parameterinformation for constructing control device frames or link controlframes. A transmit frame counter 210 generates an address pointer storedin an address pointer register 212 for the DPR 32 so that it sequencesthrough the dual port ram and retrieves all the required headerinformation for constructing a frame. For user data device frames, oncethe appropriate number of header bytes is counted/provided the transmitstate machine 202 indicates the end of the header and starts to read thetransmit FIFO. At that point the input select for the transmit frame MUX204 is changed so that the data from the transmit FIFO follows theheader information from the DPR onto the transmit frame data path.

The constructed frame moves through a buffer 214 in the data path. Dataparity control logic effects parity checking of the assembled frame.Again, the enhanced parity mechanism is implemented in the form of atransmit linear feedback shift register (LFSR) 218 and parity checker220 that are synchronized with substantially identical enhanced paritylogic on the assembler/disassembler 38 to ensure integrity of the datathrough the transmit FIFO 36.

An ESCON CRC generator 222 accumulates a frame long ESCON CRC thatcovers the entire frame. Local CRC checker logic 224 calculates a localCRC associated only with the data (DIB) portion. The local CRC is nottransmitted with the frame. The local CRC checker logic 224 accumulatesthe CRC on the data with an exclusive-or operation. The softwarecompares the local ESCON CRC with a CRC generated at the other end ofthe pipeline at the assembler/disassembler, calculated over the samedata portion.

Pacing, loopback and sequence logic (PLS) 226 implemented in thetransmit logic FPGA provides control mechanisms to implement threedifferent functions. The first is the implementation of pacing. Pacinggenerally relates to the rate of transfer of information or frames, andis specified in the ESCON protocol specification. In communicating withESCON clients, the ESCON interface according to the invention implementspacing by this control logic mechanism that introduces appropriatepacing into the transmit data stream. Secondly, loopback control logicis implemented for diagnostic purposes, and allows a pipeline's transmitdata path to be looped back to that pipeline's receiver channel.Thirdly, sequence logic provides for the generation of ESCON sequencesfor purposes of communicating at the physical level with the host ESCONI/O.

The transmitter logic 200 further includes a delimiter generator 228,that generates the delimiters for establishing the frame according tothe ESCON protocol, as known in the art. The ESCON frame delimiters aregenerated for transmitted frames by the delimiter generator, whichprovides and formats delimiters for frames in accordance with the ESCONprotocol as a function of the type of ESCON frame transfer in progress.

An output MUX 230 is controlled by the transmit state machine 202 toselectively pass the information components that constitute a frame fortransfer. Initially, the output MUX passes the starting delimiter fromthe delimiter generator 228 (on a line “SED”). Following the startingdelimiter in the frame the output MUX passes the header and the data (ona line “DD”). Then the output MUX 230 passes the CRC data which comeslast in the frame (on a line “CR”), followed only by the end of framedelimiter (on the line SED). The output MUX 230 passes ESCON protocolsequence data (on a line “SD”) from the sequence generator 226 for lowlevel ESCON communications.

The data outputs from the output MUX are buffered for transmission tothe HOTLINK transmitter 26. The transmit state machine 202 also issuescontrol signals to the HOTLINK transmitter that handles the encoding ofdata and serialization for ESCON transmission.

The line processor 68 interfaces to the transmit state machine 202 inthe ESCON protocol engine 28 through a plurality of registers in a CPUinterface 232 in the FPGA, illustrated in TABLES VI-X. A two-stagepipeline comprised of a first pipeline register 234 and a secondpipeline register 236 stage the line processor communications to the CPUinterface in the transmit state machine. The pipeline is written by thesoftware running on the line processor which defines the structure offrames by communicating with a transmitter frame register. Basically theline processor writes this register whenever there is a transmissionthat needs to be done. The CPU/line processor writes the first stageregister 234 with fields that tell the transmitter hardware what theframe will be. Ultimately, the frame content, e.g. in terms of headercontent, data length and content, etc., is defined by the informationwritten by the line processor into the CPU interface register(s). All ofthe parameters of the frame are provided in one access done by theprocessor and then the hardware takes care of the rest, and at the endgenerates an interrupt to indicate that the job is done. The framepipeline stages allow the hardware to be in progress on datatransmission and while that is happening it will allow the lineprocessor to program the next transmit operation. In this manner anyundue latency is avoided.

Again, the ESCON interface protocol engine frame transmission logic 200assembles ESCON frames, including device frames with data, for transferto the host ESCON I/O according to the ESCON format. That is, on readsfrom the ICDA data is retrieved from the disk array, stored in cache andultimately transferred to the host ESCON I/O after being processed fromthe cache by incorporating the data into ESCON frames including frameheaders as required by the ESCON protocol known in the art.

Frame headers for transmission are built in the transmit frame DPR 32 bythe pipeline processor under software control. The transmit frame DPR 32is 8 k-bytes deep and is divided into four 2-kbyte sections (slots), asillustrated in FIG. 5 b. Each slot holds only one frame at a time. LinkControl Frames are built entirely in the transmit frame DPR 32. Thismeans that both the frame header and user data or DIB portion are builtby the software in contiguous address space within the given slot. TheLink Control frame headers are built according to the structureillustrated in FIG. 5 c. Device Frames are handled somewhat differentlythan Link Control Frames. Headers for Device Frames are always built inthe transmit frame DPR 32, according to the structure illustrated inFIG. 5 d. However, subject to program control, the DIB can be builteither in contiguous transmit DPR space or the user data can be sentfrom the transmit FIFO 36.

The transmit FIFO 36 is a data buffer which collects DIB data retrievedfrom cache and then indicates when it has enough data to send out to theESCON interface. If a bit in the transmitter frame register (TABLE VI)is not set (i.e. XmitFrm.G=0), the transmit logic will retrieve the DIBportion from the transmit frame DPR 32. If the bit is set, (i.e.XmtFrm.G=1) the DIB will be retrieved from the transmit FIFO 36. Itshould be noted that all data read from the transmit DPR or Frame Store32 will be checked for ODD parity.

Once the frame is assembled based on a DIB portion either in thetransmit DPR 32 or the transmit FIFO 36, a command can be written to thetransmitter frame register (XmtFrm Register), to initiate the frametransmission.

Upon initiation of frame transmission, the delimiter generator 228 ofthe transmit machine 202 will insert the appropriate starting/endingdelimiters on the frame data. A local/data CRC is accumulated on theuser data portion of the frame retrieved from memory (by the local CRCchecker logic 224), and an ESCON CRC is accumulated by the ESCON CRCgenerator 222 for all frame data and appended onto the outgoing ESCONframe. After the frame is assembled it is passed through the output MUXor transmission multiplexor 230 and sent along with Hotlink controlsignals issued by the transmit machine 202 to the HOTLINKS transmitter26 for transmission over the ESCON link.

The XmtFrm Register in the CPU interface 232 is implemented as thetransmit pipeline with the first stage 234 and the second stage 236.This means that once a transmit operation has been initiated, thesoftware running on the line processor can prepare the next outgoingframe in the transmit frame DPR 32. New command parameters can also bewritten in the XmtFrm Register while the previous transmit is inprogress. The transmit logic monitors the pipeline and indicates whenthe transmit pipe is full by asserting a XmitPipeFull (XPF) bit in thetransmitter status register (XmtStat, TABLE VIII). The hardware alsomakes sure that the pipeline control software does not erroneouslyattempt to write the XmtFrm Register while the transmit pipe is full. Ifthis occurs the hardware will prohibit the write operation and willproceed to assert the Frame Error interrupt.

When the current frame is being transmitted, the hardware checks if anew frame is ready for transmission by checking a next frame ready bit(NxtFrmRdy) in the transmit status register XmtStat. If the bit is set,the new frame transmission will begin immediately after the currentframe is completed, except if a sequence transmission has also beenactivated, in which case, the sequence transmission takes priority.

When a frame error is detected during frame transmission, frametransmission is disabled after the current frame and the transmit pipewill not be advanced. Reading the XmtFrm register at this point willyield the command parameters for the frame that was sent in error. Ifthere is a pending frame waiting in the transmit pipe it will not besent because the pipe will automatically be flushed by the hardware.Frame transmission must then be re-enabled by clearing the frame errorcondition.

When the transmit pipeline is empty, as indicated by a bit set in thetransmitter status register (i.e. XmtStat.XPE=1), then a single write tothe XmtFrm register will start the frame transmission. A second write isnot necessary to fill the transmit pipe to allow the first frame to besent. Also, the pipeline processor software has the ability to flush theentire transmit pipeline. By asserting a control signal in thetransmitter control register (XmtCtl.FXP, TABLE VII), the pipelineprocessor software will instruct the hardware to flush the transmitpipeline.

The transmit logic 200 is configured to undertake pseudo frametransmission operations. Pseudo frame transmission means that thetransmit logic or hardware goes through all of the actions involved inframe transmission, as described hereinbefore, except actually sendingthe frame data to the ESCON Hotlink transmitter chip 26 for transmissionover the ESCON link. During this pseudo frame transmission operation thetransmit logic will instruct the Hotlink transmitter 26 to send IDLEcharacters onto the ESCON link. The transmit logic will retrieve datafrom the appropriate buffers and accumulate a data CRC. Pseudo frametransmissions are used for on-line tests to ensure hardware function bychecking CRC and status. Pseudo mode is enabled by setting a pseudo modeenable bit (XmtPLS.SDO=1) in the transmitter Pacing-Loop-Sequenceregister (See TABLE IX). It should be noted that for normal operationthe XmtPLS.SDO bit must be clear.

Communication over an ESCON link as known in the art generally requiresa capability to perform sequence transmission over the link. Thetransmitter PLS logic 226 can be instructed to send a continuous streamof sequence data. To do this the transmitter Pacing-Loop-Sequenceregister sequence bit (XmtPLS.SEQ) must be loaded by software with asequence identifier corresponding to the required sequence. If thetransmit logic is transmitting a frame when the XmtPLS register isloaded with a valid sequence, the frame transmission in progress willcomplete normally before transmitting the sequence. If the softwareinstructs the transmit logic to transmit a frame while a sequencetransmission is in progress, the frame transmission is delayed until thetime that the sequence transmission is stopped by the software, i.e. thetransmit logic is instructed to send the IDLE sequence.

A flow control mechanism is provided in the transmit logic byimplementation of a pacing count. The pacing count, effected by the PLSlogic 226, facilitates changes in the rate of transmissions onto theESCON link. The control software can specify the pacing count by writingto the register XmtPLS.PACING. The transmit logic uses this pacing countto insert IDLE characters between frames. Control software can also beused to instruct the transmit hardware to disregard the pacing count forcertain frames. Asserting a transmit frame control signal XmtFrm.EnPwill instruct the hardware to enable the pacing parameter. When pacingis not enabled/specified, a minimum of four IDLE characters are insertedbetween frames. When pacing is enabled the number of IDLE charactersinserted is computed as:4+2*(255−Pacing Count)

The local CRC checker 224 in the transmit logic can accumulate a dataCRC over all data bytes of device data frames coming from the transmitFIFO 36. The CRC value can, and should, be cleared under softwarecontrol before data transfer begins, by asserting a clear CRC bit in thetransmitter control register (XmtCtl.CCRC). The data CRC value can beread from the CRC checker by asserting a status bit in the transmitterstatus register (XmtStat.GCRC), at the end of the data transfer. If itis not cleared between frames the data CRC is accumulated over multipleframes. If the CRC is read while frame transmission is in progress, thevalue read is not guaranteed to be valid. To ensure a valid data CRC, itshould be read after the frame transmission is complete and the transmitpipe is empty. This can be checked by verifying that the XmtStat.XPE bitwas set when a FrameSent interrupt was generated. In this illustrativeembodiment, there is CRC coverage only on the user data from thetransmit FIFO 36. There is no CRC coverage on header or other data fromthe transmit frame DPR 32. Only parity protection is provided for headeror other data coming from the transmit frame DPR 32.

As mentioned hereinbefore, the assembler/disassembler mechanism (38,FIG. 3, described in detail hereinafter), has substantially identicalLFSR/enhanced parity circuitry as is implemented in the protocol engine28. The enhanced parity circuitry (FIG. 5 a, 218, 220) is used to ensuresynchronization between the protocol engine 28 and theassembler/disassembler 38, for data transfers going in either direction(i.e. through either the receive FIFO 34 or transmit FIFO 36). In thecase of data transfers from the ESCON interface according to theinvention (i.e. to the host ESCON I/O), the transmit logic and theassembler/disassembler employ the enhanced parity mechanism to ensurelongitudinal data synchronization across the transmit FIFO 36. In thisimplementation, the transmit logic alternately checks for even/oddparity for each byte that is read from the transmit FIFO 36 based on thesame pseudo-random repeating pattern discussed hereinbefore with respectto the LFSR in the receiver logic. The assembler/disassembler 38alternately generates even/odd parity on each byte that it writes to thetransmit FIFO 36 based on the same pattern. The transmit logic andassembler/disassembler will continue to run in this mode until an erroris detected or until the assembler/disassembler flushes the transmitFIFO. This enhanced parity mechanism is disabled after reset and can beenabled by asserting XmtPLS.EnDisp. If XmtPLS.EnDisp is not asserted(i.e. XmtPLS.EnDisp=0), then all data read from the transmit FIFO willbe checked for odd parity. It should be noted that all data read fromthe transmit frame DPR 32 will be checked for odd parity.

The transmit logic is configured to generate interrupts to therespective line processor in the case of four types of events, includinga Normal Interrupt; Access Error Interrupts; Frame Error Interrupts; andMachine Check Interrupts.

The Normal Interrupt on the transmit side is a FrameSent interrupt whichis an indication to the line processor that the transmit logic hassuccessfully sent an outgoing frame as instructed by the line processor.This condition is indicated by a frame sent status bit in thetransmitter status register (i.e. XmtStat.FS=1). The interrupt isenabled by asserting a control bit XmtCtl.FSEn. XmtStat.FS is clearedautomatically by the hardware when the XmtStat register is read.

Two types of access error interrupts are generated when requests aremade of the transmit logic which can not be serviced. An illegal writeinterrupt (IllegalWrt) is generated if there is an attempt to write tothe XmtFrm Register requesting that a new frame transfer be undertakenwhile the transmit pipe is full. The IllegalWrt interrupt is indicatedin the transmit status register by setting a bit XmtStat.IW=1. Anillegal request interrupt (IllegalRqst) is generated if there is arequest to start a frame transmission while an ESCON sequencetransmission other than IDLE is in progress. The IllegalRqst interruptis indicated by setting a transmit status bit XmtStat.IRQ=1.

Each of these access error conditions will also set the frame errorstatus bit XmtStat.FE=1. Before a new frame transmission can beinitiated, the interrupt and status bits must be cleared by softwareasserting XmtCtl.CFE. The Access Error Interrupts are enabled byasserting an enable XmtCtl.FEEn.

Several kinds of frame error interrupts are generated to indicate errorconditions related to various aspects of transmit frame processing.

A frame store or transmit DPR parity error, FSParErr, is generated toindicate that the transmit logic has detected a parity error whilereading the transmit frame DPR 32. Such an error is indicated by a settransmitter status register bit (XmtStat.FPE=1).

A transmit FIFO parity error, GFIFOParErr, is generated upon anindication that the transmit logic has detected a parity error whilereading the transmit FIFO 36. Such an error is indicated by a set statusregister bit (XmtStat.GPE=1).

A frame store or transmit DPR overflow condition, FrameOvf, is generatedupon an indication that the transmit logic has detected a case where anoutgoing frame has exceeded the maximum size of 1035 bytes. Such anerror is indicated by a set status register bit (XmtStat.OVF=1).

A transmit FIFO empty error condition, XmitFIFOEmp, is generated upon anindication that the transmit logic detected that the transmit FIFObecame empty while DIB data was being retrieved from it. Such an erroris indicated by a set status register bit (XmtStat.GMT=1).

If any of the above Frame Error conditions occur, the current frametransmission is ended with an abort delimiter and the transmit pipe (ifloaded) will not advance. Each of these Frame Error conditions will alsoset the Frame Error status bit (XmtStat.Fe=1). Before a new frametransmission can be initiated, the interrupt and status bits must becleared by software asserting the control bit XmtCtl.CFE. The FrameError interrupt is enabled by asserting XmtCtl.FEEn.

The transmit logic issues Machine-Check Interrupts indicative ofsignificant hardware errors, in the form of data parity errors andaddress parity errors. A data parity error, DatPE, is generated upon anindication that a data parity error was detected during a write to oneof the transmitter logic registers. This condition is indicated byXmtStat.DPE. This interrupt and XmtStat.DPE can be cleared by assertingXmtCtl.CMC. This interrupt is not maskable (i.e. it is always enabled).

An address parity error, AddrPE, is generated upon an indication thatincorrect address parity was detected during a read or write to/from oneof the transmit logic registers. There is no status bit to indicate thiscondition. This interrupt can be cleared by asserting XmtCtl.CMC. Thisinterrupt is not maskable.

On Power-Up and after Reset, the transmit logic comes up in thefollowing state:

-   -   transmit logic is transmitting the Offline sequence;    -   XmtStat.XmitPipeFull=0;    -   XmtStat.XmitPipeEmpty=1;    -   Enhanced Parity Disparity checker is OFF;    -   XmtPLS.EnDisp=0;    -   All interrupts are disabled except for Machine-check.

The ESCON interface transmitter and receiver logic according to theinvention provide a mechanism by which pipeline processor software caninform the protocol engine hardware that it is busy and, as a result,the hardware will hold off any connection requests from the host. Inthis scenario, the receiver logic 100 will look for any incomingconnection frames from the host ESCON I/O and then instruct thetransmitter logic 200 to respond to these connection frames by returninga link-busy frame to the host. It should be noted that the receiverlogic will not buffer the incoming connection frames and will notinterrupt the pipeline processor. This behavior will continue until thesoftware instructs the hardware to return to normal operation.

A BUSY bit is provided in the receiver mask/miscellaneous register(RcvMM Register, TABLE III). This bit is READ/WRITE. Setting this bittells the hardware to begin returning link-busy frames in response toincoming connection requests. Clearing this bit tells the hardware toreturn to normal operation. The CPU interface is asynchronous to theESCON clock. Therefore, there is no guarantee that once the BUSY bit isset, frame interrupts will cease. There may be cases when one more frameis received and buffered after the BUSY bit is set.

Assembler/Disassembler Mechanism

Referring now to FIGS. 3 and 6, the assembler/disassembler mechanism 38is interposed between the FIFOs (transmit and receive) 34, 36 and theSDIO bus 40 connected to the buffer DPR 42 that receives user data goingto/from Global Memory (GM), through the EDAC. The primary role of theassembler/disassembler 38 is the actual assembly and disassembly of datamoving along a pipeline to/from GM. On a write to GM the user data thatcomes from the receive FIFO 34 is 9 bits wide of which 8 bits is data,plus one enhanced parity bit. The receive FIFO data is assembled by theassembler/disassembler on a write into 64 bit memory words plus 8 bitsof parity. On a read from GM, the 72 bit memory data, again a 64 bitmemory word and 8 bits of parity, is disassembled by theassembler/disassembler 38 into 9 bit wide data, an 8 bit data byte plusone bit of parity, for storage in the transmit FIFO 36. Theassembler/disassembler 38, in this illustrative embodiment functions tohandle assembly and disassembly selectively for two pipelines. That is,each assembler/disassembler 38 services two pipelines each comprised ofrespective transmit and receive FIFOs, respective transmit and receiveframe DPRs, respective protocol engines and respective transmit andreceive HOTLINK devices.

The assembler/disassembler 38, as illustrated in greater detail in FIG.6, comprises a FIFO machine 300 that interfaces with the receive FIFO 34via a 9 bit wide receive FIFO data bus (RFDAT) and with the transmitFIFO 36 via a 9 bit wide transmit FIFO data bus (TFDAT). The FIFOmachine 300 monitors the status of the FIFOs and determines if it shouldbe reading the receive FIFO 34. If the receive FIFO 34 is empty, on awrite to GM, the FIFO machine will stop reading. If the FIFO machine 300is writing the transmit FIFO 36, on a read from GM, and the transmitFIFO becomes full, the FIFO machine will stop writing the transmit FIFO.The FIFO machine 300 basically handles all the protocol for transferringdata in and out of the FIFOs 34, 36.

The assembler/disassembler mechanism 38 includes a synchronous memory302 and associated counters 304. The FIFO machine 300 also handles theprotocol for transferring data in and out of the synchronous memory 302.On a write to GM, the FIFO machine 300 assembles 72 bit data andtransfers it to the synchronous memory 302 through a memory multiplexor(MUX) 306 to an input data bus (IDAT). The 72 bit global memory data isthen available to be transferred out of the synchronous memory 302 overa 72 bit output data bus (ODAT) ultimately destined for the SDIO bus 40.On a read from GM, the synchronous memory 302 receives the 72 bit globalmemory data over the SDIO bus 40 through the memory MUX 306. Thesynchronous memory 302 effectively disassembles the global memory databy transferring it to the FIFO machine 300 a byte at a time over a ninebit wide bus (ZDAT), transferring 8 bits of data and one parity bit.

The assembler/disassembler 38 also contains a counter mechanism thatcounts the amount of data that is read into and out of the FIFOs. Whenthe data transfer starts, the line processor can set how much data willbe transferred to memory and from memory in and out of the FIFOs, whichallows the entire data transfer to effectively run independent of theline processor. When the transfer count is exhausted/reached, thepipeline hardware can flush the data to memory and initiate end oftransfer checks. The data can be tagged with the additional errordetection and information, i.e. control status information including CRCinformation can be updated and/or appended to the data progressingthrough the pipeline. This effects an additional level of errordetection by ensuring at the assembly/disassembly part of the transferthat all of the expected data was transferred into or out of the FIFO(in agreement with a count in the Middle Machine described hereinafter).Other mechanisms in the pipe, e.g. the Middle Machine, can then be mostexpeditiously informed whether a complete transfer was effected.

The counters machine 304 implements the counter mechanism in the form ofpointers for the synchronous memory to facilitate transfers in and outof the synchronous memory 302 (i.e. into and out of the FIFOs 34, 36 viathe FIFO machine 300). The counters machine 304 includes a read pointer(RPTR) and a write pointer (WPTR) issued to the synchronous memory as afunction of the operation being performed. The read pointer and thewrite pointer are each actually used in two instances on a read and twoinstances on a write. The read pointer is used when data is beingtransferred from the SDIO bus to the synchronous memory and also whendata is being transferred from the synchronous memory to the FIFOmachine. The write pointer is used when data is being transferred fromthe FIFO machine to the synchronous memory and again when data is beingtransferred from the synchronous memory to the SDIO bus.

In the assembler/disassembler 38, the synchronization between the memoryports (i.e. read port(s): IDAT/ZDAT, and write port(s): IDAT/ODAT) hasto be tightly controlled. The addressing to the read port has to be verytightly coupled to its counter/pointer and the write port has to be verytightly coupled to its counter/pointer in order to maximize performance.So the counters themselves and the pointers that they define arespecialized, as a read pointer or a write pointer, as a function oftheir selective operation in reading data from the SDIO into the synchmemory or reading data from the synch memory into the FIFO machine, orwriting data from the FIFO machine to the synch memory or writing datafrom the synch memory to the SDIO. The pointers get loaded with initialvalues at the beginning of the data transfer. The initial values and theoperation being performed, i.e. a read from GM or a write to GM,determines whether the read pointer is acting as a pointer for readingdata from the SDIO into the synch memory or reading data from the synchmemory into the FIFO machine, and whether the write pointer is acting asa pointer for writing data from the FIFO machine to the synch memory orwriting data from the synch memory to the SDIO bus 40.

The assembler/disassembler 38 includes an SDIO machine 308 that isprimarily a protocol machine to run the protocol on the synchronous SDIObus 40. The SDIO machine 308 communicates with the lower machine 46,described in detail hereinafter, to handle data transfers to the globalmemory or from the global memory. The protocol in both directions issubstantially the same, but the placement of data is different.

A processor machine 310 in the assembler/disassembler 38 incorporatesregisters that provide a pathway for the respective line processor toget access into the data string, and allow the processor to access theGM, effectively replacing the known MDR pipeline. The processor machine310 also implements a processor interface, in the form of anassembler/disassembler control register illustrated in Table XI, and anassembler/disassembler status register illustrated in Table XII. Theprocessor interface in the processor machine(s) 310 enables lineprocessor(s) 60 and/or the service processor 70 to communicate with achannel to/from GM.

The assembler/disassembler command register (Table XI) essentiallyincludes information indicating the number of bytes involved in atransfer, and a bit that is set by the line processor to indicatewhether a transfer is a read or a write (DIR). A command bit must be setto enable any transfer (EnXfr).

The assembler/disassembler command register includes bits relating tothe type of error checking to be performed on data passing between theassembler/disassembler 38 and the FIFOs 34, 36. A command register bitcan also be set to indicate whether enhanced parity (EnDSP) isimplemented on transfers.

The assembler/disassembler mechanism 38 in each pipeline implements inthe FIFO MACHINE a substantially identical enhanced parity mechanism asimplemented in the protocol engine for use when receiving data fromFIFOs (on a write operation to GM), and transferring data to FIFOs (on aread operation from GM). The enhanced parity mechanism in theassembler/disassembler, like the one in the protocol engine, isconfigured to generate a pseudo-random number for each data byte.

As discussed, the FIFO's are 9 bits wide with the ninth bit used asparity. However, using the ninth bit as parity does not protect againstgetting the same byte of information twice. In addition to the paritythat is written into each byte, the parity is selected as even or oddbased on the order of the bytes. A linear feedback shift register (LFSR)is used to generate a pseudo random pattern. The LFSR, in both theassembler/disassembler 38 and the protocol engine 28 is a hardwareimplementation of a mathematical polynomial that is implementationspecific. The pseudo random pattern, as described hereinbefore, is usedto select whether the parity is even or odd for that particular bite.

This enriched parity is an additional check which protects against theFIFOs making a gross internal error, perpetuating an addressing error orjust sending the wrong piece of data. If the pseudo random pattern doesnot match it will show up as a parity error and the transfer can bestopped or the data can be transferred on another channel. The enrichedparity provides a way to detect if there is a failure along a pipeline,and facilitates invocation of diagnostics to determine the source of thefailure.

The enhanced parity generated by the protocol engine 28, accompanyingdata transferred from the FIFO (on a write to GM) is exclusive-ORed withLFSR data generated by the enhanced parity circuitry in theassembler/disassembler 38 (which effectively reproduces the originalparity stored with the data in the FIFO). The exclusive-OR operationeffectively checks the FIFO parity information against theassembler/disassembler generated parity.

If the pointers in the FIFOs become disjoined so that data isessentially being read out-of-sequence from the FIFOs, the conditionwill be detected in a significant number of the cases as it shows up asa parity error in either the protocol engine or theassembler/disassembler. For example, addressing errors in the FIFOs willmanifest themselves as parity errors in the assembler/disassembler 38 ifthe data coming from the protocol engine 28 comes through the FIFOs withopposite parity than what is computed/expected at theassembler/disassembler 38 (on a write to GM). Likewise, addressingerrors in the FIFOs will manifest themselves as parity errors in theprotocol engine 28 if the data coming from the assembler/disassembler 38comes through the FIFOs with opposite parity than what iscomputed/expected at the protocol engine 28 (on a read from GM).

The enhanced parity is in addition to the security provided by the CRCthat is accumulated on both sides of the FIFO, as describedhereinbefore. An assembler/disassembler command register bit can be setto indicate whether a CRC is to be tacked onto the end of the datatransfer. Eight bits in the assembler/disassembler command register(Acrc0-Acrc7) indicate the CRC word on the end of the data transfer. Theassembler/disassembler calculates a running CRC on the data stream. Thisdata is byte wide and can be inspected in the assembler/disassemblerstatus register (Table XII). The CRC is only reset by direct command(reset CRC). The CRC is never passed into the FIFO stream. The CRC canbe automatically inserted into the data stream on a write (With CRC), orzeroed with an additional stream data byte on a read. The zeroing of theCRC can allow the automatic checking of the CRC by producing a resultantzero in the CRC on a successful CRC check.

The command register CRC bits can be used to provide a seed value forthe CRC. That is, the CRC value can commence from a starting point otherthan zero. It may be desirable for each processor on an interface cardto seed the CRC with a different value so that distributed seed valuesprovide a difference in the CRC that facilitates determination of theline that is doing the transfer.

The assembler/disassembler 38 essentially handles all datapathtransactions between the protocol engine 28 transmitter logic, theprotocol engine 28 receiver logic, the service processor SCSI ASIC 71,the line processor data bus and the buffer Dual Port RAM 42. Theassembler/disassembler 38 for each data pipeline or channel is connectedbetween the FIFOs 34, 36 and the buffer DPR 42 (and an MDR between theSCSI ASIC 71 and the buffer DPR 42 for the Service Processor 70).

There are four possible data paths for utilization of eachassembler/disassembler 38: the ESCON data path between the protocolengine 28 and the DPR 42 (referred to as the ADT path for “AutomaticData Transfer”); two DMA data paths for the line processor in each lineto directly access DPR for Direct Memory Access operations (DMA0 ANDDMA1), and one data path for the line processor to access the top of theFIFO (Top of FIFO).

Generally, Automatic Data Transfer (ADT) is accomplished by programmingusing the Middle Machine's Address Pointer, Command, and StatusRegisters (discussed in detail hereinafter), and programming using theESCON receiver registers (Tables II-V) and/or the ESCON transmitterregisters (Tables VI-X). The transfer will be initiated when theassembler/disassembler command register (Table XI) is programmed withthe transfer count number (XC[0-15], in bytes, the direction of transfer(DIR), and when the Enable Transfer (EnXfr) bit is set.

The assembler/disassembler according to the invention is configured toassemble or accumulate burst data, as opposed to moving one piece ofdata at a time, for transfer. Thus when a threshold quantity of data isaccumulated for an efficient transfer, the assembler/disassembler passesthe burst of data along the pipeline.

An ADT Write is a transfer from the ESCON protocol engine (through thereceive FIFO 34 and assembler/disassembler 38) into the DPR 42. In termsof functionality, the assembler/disassembler 38 assembles the data intomemory words (8 data bytes plus one parity byte) once data is writtenfrom the ESCON receiver logic in the protocol engine 28 to the receiveFIFO 34. When the assembler/disassembler 38 has assembled 8 memorywords, it will send a request for transfer to the Middle Machine 48.When the requesting pipe receives an acknowledge from the Middle Machinearbitrator (on the Lower side), the assembler/disassembler 38 will beginthe data transfer burst to DPR 42.

An end of transfer signal (EOT) will automatically be sent with the lastmemory word transferred indicating to the Middle Machine 48 that thecurrent burst from the assembler/disassembler 38 to the DPR 42 iscomplete. EOT indicates that the assembler/disassembler transfer counthas expired. The Middle Machine will flush all remaining data from theDPR 42 to Global Memory upon receiving the EOT signal. Theassembler/disassembler will continue reading data from the receive FIFO34 and bursting the assembled data to DPR until its transfer counter hasexpired. At this time, a Command Complete (CC) indication will bewritten to the assembler/disassembler status register (Table XII),indicating that the assembler/disassembler is idle.

An ADT Read is a transfer from the buffer DPR 42 to an ESCON channel.The assembler/disassembler 38 will request a transfer from DPR to theassembler/disassembler. Once the requesting pipe has received anacknowledge from the Middle Machine arbitrator (Lower side), theassembler/disassembler will begin the data transfer burst from the DPR.Data is disassembled and written byte-wise to the transmit FIFO 36. Theassembler/disassembler 38 will continue reading memory words from theDPR 42 and writing them byte-wise to the transmit FIFO 36 until theassembler/disassembler's transfer counter has expired. At this point, anassembler/disassembler interrupt will be generated to the lineprocessor, and Command Complete will be written to theassembler/disassembler status register.

There are two DMA (MDR) pipes per line processor assembler/disassembler:DMA0 and DMA1. The DMA pipes effectively permit the line processorsdirect access to GM for writing and reading through the buffer DPR 42.Line processor DMA Writes are handled by programming the MiddleMachine's Address Pointer, Command, and Status Registers, writing theassembler/disassembler's DMA0 (or 1) MDR LSW (Least Significant Word)Register (not shown) followed by a write to the DMA0 (or 1) MDR MSW(Most Significant Word) Register (not shown). Writing to the MDR MSWRegister causes the assembler/disassembler to request for transfer tothe Middle Machine. When the pipe receives an acknowledge, theassembler/disassembler will transfer the data to the buffer DPR 42.There are two end of transfer (EOT) addresses in theassembler/disassembler per DMA pipe. Writing to a basic EOT addressindicates that the assembler/disassembler has completed the lineprocessor DMA transfer. Writing to an MDR-with-auto-EOT address in theassembler/disassembler will force EOT to be automatically asserted whentransferred to the MM (along with the data). This will cause the MM toinitiate a transfer between the line processor and Global Memory foreach DMA transfer.

DMA reads are initiated by programming the Middle Machine with theaddress, direction of transfer, and length. Then, the data can betransferred to and read from the corresponding assembler/disassemblerMDR LSW and/or MSW register. A read from the MSW register will fetch thedata and return the MSW of the data, and a read from the LSW registerwill return the LSW from the last fetch. The LSW need not be read if theLSW of the data is not needed. At the completion of the line processorDMA read, the Middle Machine will return an interrupt that may occurbefore the last data is read from the MDR register. The line processorcan receive errors in the form of parity errors, Middle Machineexception, or timeout. A timeout is generally the result of an error inthe Middle Machine—either a reported error or the transfer length wasincorrectly programmed.

A Top of FIFO (TOF) function, implemented using a Top of FIFO registerin the assembler/disassembler, allows the software running on the lineprocessor to pick off data from the head of the received stream (i.e. atthe top of the receive FIFO). Similarly, data can be placed into thetail transmit stream (i.e. at the bottom of the transmit FIFO), via aBottom of FIFO register implemented in the protocol engine transmitterlogic. The positioning of this data is only relevant while theassembler/disassembler is idle. Accessing the TOF register while theassembler/disassembler is operating may produce undesired results. Ifthe transmit FIFO is full when data is written to the TOF, the data isdropped. If the receive FIFO is empty when data is read from the TOF,the data is returned invalid. If disparity or enhanced parity isenabled, a write to full FIFO or read from an empty FIFO will causesubsequent parity failures in the FIFO data stream if the FIFO is notflushed to synchronize the stream.

A PAD bit in the assembler/disassembler command register (Table XI) canbe set to nullify the transfer of data to and from the FIFOs 34, 36.This feature can be used to generate a CRC calculation or to check overa Global Memory address range, or to fill a range in Global Memory withzero data. For example, the assembler/disassembler 38 on a writeoperation can fill or pad a portion of the data transfer with a padbyte. Similarly, the whole memory or a block of memory can be filledwith a known value.

The pad feature is important for formatting a CKD frame according to aformat command from the host, which requires that the data field beformatted to a given size. If the line processor during a writeoperation needs to write that data field with less than what wasformatted, the ICDA must accommodate the transfer by filling up the restof the data field with zeros. Such an operation is defined by the IBMESCON stack called the “Mstack”. To accommodate such an operation thesoftware running on the line processor can be configured to program theassembler/disassembler with the size of the data transfer and then oncethat data transfer completes, it will then fill the remainder of therequired data with pad data.

In such a transfer the pad operation facilitates determination of a CRCon the partial transfer, i.e. on the actual customer data, by nottransferring the CRC at the end of the actual data. Rather, theassembler/disassembler 38 stores the CRC that was transferred at the endbut it does not transfer it to memory. When the transfer is completed tomemory with the pad data to fill the required data field, theassembler/disassembler will transfer all the zeroes, which do not affectthe CRC, and then at the end of the required size data block it willappend the CRC.

On a read the pad operation may also be implicated, as the lineprocessor may not ask for the whole record, for example it may requestonly the first 1 k of the record. In order to do the error detection,the CRC is at the end of the record. Thus after the 1 k is read, therest of the data is screened through just to calculate out the CRC tomake sure that, overall, the CRC is right. That is, on a read operationwhen the pad bit is set, the assembler/disassembler keeps reading thedata beyond that which is requested by the line processor, but it doesnot put it into the FIFO. The assembler/disassembler just calculates upthe CRC and checks the CRC at the end to return good or bad status basedon how it ended up. In this manner the assembler/disassembler does a lotof the processing that in known systems is done all by the CPU. In orderto do this in an existing system, the CPU typically has to do DMA readsand go through the entire memory itself to calculate the CRC and makesure it was right for the rest of memory to the end record. The padoperation in the assembler/disassembler according to the inventionoffloads some of this operation into the hardware.

The Buffer Dual Port Ram

The buffer dual port ram (DPR) 42 (best illustrated in FIG. 3), isconfigured to receive data for buffering read operations from and writeoperations to the GM. Data is transferred between theassembler/disassembler and the buffer DPR efficiently with a maximumburst size of sixteen, 64 bit memory words. The DPR is used to bufferand speed-isolate the different transfer rates of the backplane to/fromGM and various data paths on the ESCON interface circuit board. Datatransfers between the assembler/disassembler and the buffer DPR passthrough Error Detection And Correction circuitry (EDAC) 44 implementinga modified hamming algorithm as known in the art. In this illustrativeembodiment, the DPR is divided into sixteen 2 k entities. Each of those2 k are divided into 128 bytes of 32×8 bytes.

Data Transfer State Machines

The plurality of state machines comprising the data transfer statemachines are arranged into three functional units, the Upper Machine 50,the Middle Machine 48 and the Lower Machine 46, as illustrated in FIGS.3 and 7. These state machines facilitate movement of user data betweenthe buffer DPR 42 and the Global Memory (GM, not shown).

Generally, the Middle Machine 48 controls all data movement to and fromthe GM. Although not directly in the data path, it is responsible forcoordinating control between the various elements that comprise the datatransfer channels as described. The Middle Machine is interconnected tothe Upper Machine 50 and Lower Machine 46, and to various global memoryand line processor interface, address and control elements.

The Middle Machine 50 provides control and coordination between theUpper and Lower sides of the buffer DPR 42. The Lower Machine 46connects to the assembler/disassembler mechanism 38 of each pipe andcontrols the EDAC 44. The Upper Machine 50 connects to the backplane,which in turn connects to Global Memory. The actual data transfersbetween the buffer DPR 42 and GM are controlled by the Upper Machine 50and Lower Machine 46.

One the Lower side, the Lower Machine handles the handshake protocolbetween the assembler/disassembler, the EDAC and the DPR. When anassembler/disassembler needs to transfer data, it sends a request to theMiddle Machine. The Middle Machine arbitrates among the plurality ofLower side requesters, and begins a Lower grant cycle. The MiddleMachine provides the number of the pipe that won the arbitration (the“won pipe number”), along with a Dual Port RAM pointer to the LowerMachine. When the Lower Machine accepts this data, it sends anacknowledge back to the assembler/disassembler which won thearbitration, and begins a data transfer between theassembler/disassembler and the DPR, through the EDAC. Upon completion ofthe transfer, status and ending pointer information are returned to theMiddle Machine.

Similarly, the Upper Machine implements the backplane protocol toaccomplish data transfers between the Global Memory connected to thebackplane and the buffer DPR on the ESCON interface according to theinvention. The Middle Machine determines a pipe that needs servicing.When the Middle Machine finds a candidate it provides the won pipenumber along with a DPR pointer to the Upper Machine. The Upper Machinerequests backplane access and provides the address, command and transferlength to the backplane. When the data transfer phase begins, the UpperMachine manages the DPR address and control signals. Upon completion ofthe transfer, status and ending pointer information are returned to theMiddle Machine Status Register.

The Middle Machine 48, as generally described provides control andcoordination between the Upper Machine 50 effecting the transferprotocol between the buffer DPR 42 and the backplane (GM), and the LowerMachine 46 effecting the transfer protocol between the buffer DPR 42 andthe assembler/disassembler 38. As described, the Middle Machine isresponsible for coordinating transfers between the upper and lower sidesof the buffer DPR 42.

The Middle Machine 48 itself comprises a register file 100, and aplurality of state machines, as illustrated in FIG. 8. The register file100 contains four registers per pipeline. The registers include aprimary address register Table XIII, a secondary or mirror addressregister Table XIV, a command register Table XV, and a status registerTable XVI. Each line processor effecting a transfer has its respectiveset of registers in the register file that will be affected by theprocessor undertaking a transfer.

Generally, control over the data transfers is effected in theseregisters in the register file 100 in the Middle Machine. The transferswill ultimately be carried out by upper and lower machine protocols,however, these four registers are the means of communication between themiddle machine and the upper and lower machines for controlling the datatransfer. The register file in the Middle Machine, for each pipe,maintains a transfer counter which is used to pace a transfer and tocalculate appropriate backplane burst sizes. The Middle Machine registerfile is further used for error management. When an error occurs during atransfer (either Upper or Lower), the Middle Machine will collect statusfrom the Upper and Lower machines, appropriately post the errorcondition in the Status Register for the pertinent pipe, and disable thecurrent transfers. There are some ancillary registers, not shown, usedfor interrupt status or chip configuration, which is done on a globalbasis.

In an ESCON write situation data is transferred from the host ESCON I/Oto global memory. The first thing the software does is clear the statusregister for the appropriate pipe by writing zero to the statusregister. Secondly, the software programs the middle machine's primaryaddress pointer with a global memory address. It determines that thereis a portion of memory allocated to the data transfer and places thataddress in the primary address pointer. The second part of theinformation provided to the Middle Machine may be a mirror pointer.

When the address is written into the address register the hardware makesa copy of the lower 5 bits of the address register into the lower 5 bitsof the upper and the lower pointer which are contained in the statusregister. Therefore, bits 25 through 31 of the address pointer arecopied into the upper pointer bits 3 through 7 and lower pointer bits 3through 7. Then the most significant bits of the upper and lower pointerare set to zero. When all the transfers are done the hardware will checkto make sure that the final lower 5 bits of the address pointer, thefinal 5 bits of the upper pointer and the final 5 bits of the lowerpointer are all the same. If they are not all the same then somethingmiscounted, too high or too low, or something is still not complete.This provides a simple hardware check which is used as a double checkagainst counting faults, hardware faults, misclocks or the like.

The command register in the register file 100 is written to contain atransfer length which is the number of double words, i.e. 72 bitentities which are to be transferred. It will also contain a set ofcommand bits which will describe things which are common to the globalmemory protocol such as whether it is a read or a write, whether or notit is a lock operation or whether it is a service cycle. There aretwelve command bits, bits 20 through 31. These writes to the registerfile 100 in the middle machine 48 prepare for the transfer.

Again, the Middle Machine is the entity which controls all data movementto and from the global memory. The Middle Machine provides control andcoordination between the Upper and Lower sides of the DPR.

On the lower side of the DPR, when an Assembler/Disassembler 38 needs totransfer data, it sends a request to the Middle Machine. The MiddleMachine arbitrate among the 16 Lower side requesters, and based on itspointer processing logic, it begins a Lower grant cycle. The MiddleMachine will provide the won pipe number along with a DPR pointer to theLower Machine. When the Lower Machine 46 accepts this data, it sends anacknowledge back to the assembler/disassembler 38 which won thearbitration, and begins a data transfer between anassembler/disassembler and the DPR. The Lower Machine 46 will providethe DPR address and monitor the EDACs to determine if there are anyparity errors (on a write), or any single or Multi-bit errors (on aread). Upon completion of the transfer, status and ending pointerinformation are returned to the Middle Machine. The ending status andpointer information are stored back into the status register in theregister file 100.

Very similarly, on the upper side of the DPR the Upper Machineimplements the backplane protocol to accomplish data transfers betweenglobal memory and the ESCON board. The Middle Machine sequentially pollsthe internal Command registers of all 16 pipes to determine if any pipeneeds servicing. This is indicated by the Command Register having boththe IEC and EC bits set. When the Middle Machine 48 finds a candidate(and the Lower/Upper pointer processing logic allows the transfer), theMiddle Machine will provide the won pipe number along with a DPR pointerto the Upper Machine 50. The Upper Machine will then request backplaneaccess, and provide the address, command and transfer length to thebackplane. When the data transfer phase begins, the Upper Machine 50manages the DPR address and control signals. Upon completion of thetransfer, status and ending pointer information are returned to theMiddle Machine. The ending status and pointer information are storedback into the Status Register for the transferring pipe in the registerfile 100.

The Middle Machine maintains the transfer counter in the register file,which is used to pace a transfer and to calculate appropriate backplaneburst sizes. With this information, and in conjunction with hardware Endof Transfer (EOT) signaling, the Middle Machine is able to handle anyburst size from 1-32 Dwords, misaligned memory transfers, and “hanging”EOT cycles (auto-EOT). The Middle Machine is further responsible forerror management. When an error occurs during a transfer (either Upperor Lower), the Middle Machine will collect status from the Upper andLower machines, appropriately post the error condition in the StatusRegister, and disable the current transfers. Errors which invoke thislogic include Parity Errors, EDAC error, and count misses.

One of the responsibilities of the Middle Machine is to manage thebackplane address along with the Upper and Lower pointers. As previouslyindicated, the Dual Port RAM is used to buffer and speed-isolate theGlobal Memory transfers to and from the on-board systems. This isaccomplished through a pointer management algorithm. FIG. 9 illustratesthe relationship between the Global Memory address and the pointeraddresses. In this illustrative embodiment, the DPR 42 is segmented suchthat each pipe receives 8 buffers of 256 bytes apiece. This translatesinto 8 full backplane bursts of 32 Dwords. When a line processor writesan address into the Address Pointer Register, the hardware mirror-writesthe lower 5 bits of address into the Status/Upper & Lower PointersRegister. The three most significant bits of the Upper/Lower pointers(P0-P2) are written to ‘0’s. This provides the beginning point in DualPort memory for a transfer to occur. If the Global Memory address bitsA27-A31 are all ‘0’, the transfer is considered to be aligned (since theaddress lies on a natural 32-Dword Global Memory “page”). If any of the5 LSBs are nonzero, then the first transfer will be a misalignedtransfer. The Middle Machine will automatically determine this, andbased on the programmed Maximum Burst Size for the backplane, it willcalculate the appropriate beginning burst size to align itself onto anatural Global Memory page. The three MSBs of the pointers are used tomove through the 8 Dual Port RAM segments. As a transfer progresses, theaddresses are incremented. For transfers longer than 2 k, the pointerwill wrap.

Given the possibility of pointer wrapping, there must be some managementto prevent data over or under runs. The Middle Machine uses the 3 MSBsof the Upper and Lower pointers to determine whether a transfer shouldbegin or not, thereby preventing data corruption. FIG. 10, illustrateshow the pointer management scheme operates. Assume that the MiddleMachine has been appropriately programmed to initiate a Global Memoryread operation via a DMA pipe. The Upper and Lower pointers beginaligned to the same segment in the DPR. The Middle Machine will notgrant any Lower requests yet, since it knows that the data requested isnot yet in the DPR. This prevents the assembler/disassembler fromretrieving “garbage” data from the DPR. This is depicted in the leftmostpart of the diagram of FIG. 10. Once the Upper Machine transfers atleast one segment-worth of data into the DPR, then the Middle Machinewill allow the Lower Machine to operate. This is shown in the centerpart of the diagram of FIG. 10. If the Lower pointer “catches up” withthe Upper pointer (for example, the Global Memory stalled), then theMiddle Machine will again prevent Lower requests from being granted.

Note that the same logic holds true for the Upper pointer. Since abackplane transfer is initiated immediately after a valid Read commandis written into the Command Register, it is possible that the 8 segmentsof the Dual Port RAM may fill before a single word is read from theLower side. The Middle Machine will pause any transfers on the Upperside until a full segment has been freed by the Lower side.

Write operations work very much the same, except that the “pointerchasing” has turned around, i.e. the Upper pointer lags the Lowerpointer.

Note that the Upper and Lower pointers can be written by the CPU intothe Status Register. This allows several levels of flexibility. Fordiagnostic purposes, count miss errors can be induced by forcing thelower 5 bits of the pointers to a value other than the lower 5 bits ofthe Global Memory address. For on-board memory testing, the CPU can movethe Upper pointer to a segment other than that of the Lower pointer,thereby allowing Lower Machine access without initiating backplanetransfers. For emulation purposes, it may be useful to manually move theLower pointer to various places in the DPR. For example, if the addressoffsets of a table in the DPR were known, they could be used to programthe Lower pointer thereby eliminating the need to repeatedly read thememory via an MDR until the data of interest is transferred.

Generally, the middle machine is responsible for arbitration. It isresponsible for determination of control. It does that algorithm work todetermine whether or not a particular requesting pipe should be granted.It then passes a packet of information to the lower machine for it to dowork. The lower machine will do the work but it also has to report backstatus to the middle machine. The lower machine tells the middle machinewhether or not there were errors that occurred during the transfer[determines errors] and reports back the ending pointer. Because thelower machine has to transfer one or more words of data to the DPR itcontains the counter which increments the address to the DPR. Therefore,the lower machine sends a packet of information back to the middlemachine containing status information. There is another handshake whichoccurs between the lower machine and the middle machine after thetransfer is completed. In that handshake the lower machine indicatesthat it is done with its transfers and the middle machine will requestthat information. The lower machine will then transfer a packet ofinformation to the middle machine and then the lower machine is againfreed. After the transfer the lower machine is free to do anothertransfer.

The middle machine is architected to do parallel processing. The middlemachine is described hereinafter and illustrated in more detail in FIGS.8 and 11-15. The middle machine comprises six state machines, inaddition to the register file 100. The six state machines are segregatedinto an upper side and a lower side.

The lower side of the middle machine consists of three different controlmachines, the lower arbiter 102, the lower control machine 104 and thelower completer 106. They each run simultaneously and somewhatindependently. They each maintain indexes into the register file 100.The lower arbiter 102, which is a state machine the functionality ofwhich is illustrated in FIG. 11, initially is in an idle mode 108 andlooks for pipes requesting a transfer. It extracts data from theregister file 100 of the middle machine and determines whether or not aparticular pipe has to transfer 110 from the lower side into DPR. If atransfer is indicated, the lower arbiter 102 passes 112 relevantinformation from the register file 100 to the lower control machine 104.Thereafter, the lower arbiter 102 waits for a handshake 114 from thelower control, and then returns to the idle mode 108 to look for pipesrequesting a transfer 108.

The lower control machine 104 implements the first part of the handshakebetween the lower machine 46 and the middle machine 48. It waits in anidle state 116 for information from the register file to be passed fromthe lower arbiter 102. The lower control machine 104 passes a packet ofinformation to the lower machine by putting the information 118 on alower machine pointer bus (LMPTR), and issuing a signal 120 telling thelower machine to go (LMGO). Then the lower control machine 104 waits 122for an acknowledge (LMGAC) from the lower machine 46 indicating that thepacket of information starting the transfer was received. When lowercontrol machine 104 receives the acknowledge it will pass the packet ofinformation to the lower completer 106, and send a release flag back tothe lower arbiter 102.

The lower completer 106, the functionality of which is illustrated inFIG. 13, is waiting in an idle state 124 for an indication that thelower control machine has some activity. It receives a signal !LMNXTfirst from the lower control machine 104 and then a packet from thelower machine 46. The packet from the lower machine is the informationregarding the transfer, from the lower machine 46 to be put in theregister file 100. The lower completer requests the register file 126and when it can access it, it puts 128 the information from the lowermachine into the register file. Thereafter it issues a next acknowledgesignal (LMNACK) 130 to the lower machine to complete the handshaking.The lower completer 106 then returns to the idle state 124.

In the process of accessing register file information, the informationis checked to effect operations indicated by the nature of the bits setas described hereinbefore with respect to the register file contents.Upon completion, the packet of information that is transferred betweenthe lower machine and the middle machine contains the status and thepointers. The lower machine reports back to the middle machinerequesting pointer information. The lower machine captures parity errorsthat occurred while the transfer is in progress that would be reportedby the EDAC. In the read case the lower machine determines whether thereare single or multi-bit errors based on the ECC code. There are protocolerrors that can be reported if the lower machine detects something inerror on the lower SDIO transfer protocol. The lower machine can abortthe transfer and report back status regarding the abort.

The lower completer receives a packet of information back from the lowermachine. It is responsible for putting the information back in theregister file memory inside the middle machine. Therefore, the lowercompleter takes the pointer and puts it back into the lower pointerlocation of the status register. The lower completer will then look atthe error bits. If there were errors that were reported by the lowermachine the lower completer will set the appropriate error flags in thestatus register. Depending on the severity of the error it will stop thetransfer. If the lower completer has to stop the transfer it will alsowrite to the command register.

Referring to FIGS. 8 and 14-16, the upper side of the middle machine 48comprises a plurality of state machines including an upper arbiter 108,an upper control machine 110 and an upper completer 112. Rather thanlooping in an idle state, the upper arbiter, a state machine illustratedin FIG. 14, looks 132 at the command registers of the pipes to determinewhat actions to take. Part of its function is to walk through commandregisters of the pipes in the register file 100, continuously lookingfor commands that have to be executed. The upper arbiter looks at theInternal Enable Channel (IEC) bit 21 and the Enable Channel (EC) bit 31.The upper arbiter is continuously looking through 16 memory locationswhich contain 16 different command registers looking for those commandregisters that contain the IEC and the EC bits set. When both of thosebits are set that will indicate to the upper arbiter that there is workto do in this pipe.

Another part of the upper arbiter function is looking to make sure thatno error bits are set. If the fatal error [FE] bit is set then the upperarbiter ignores that pipe.

When the upper arbiter 108 finds a pipe with the IEC and the EC bits setit will initiate a protocol based on pointer management, describedhereinbefore, to determine whether or not this is a pipe which isallowed to be granted. The upper machine does a pointer managementfunction to determine whether or not a transfer can now be made from theDPR to the backplane. The upper machine is looking to see if there isenough data in the DPR to do a complete burst across the backplanewithout underflowing. An underflow occurs when there is not enough data,e.g., if 32 double words of data were expected but there are only 16double words of data in the DPR. There is nothing in the protocol tostop the burst so there must be enough data to satisfy the full burst.

The upper arbiter 108 then makes sure that the pointers are such thatthere will not be an overlap case. It looks to see that the segmentaddresses for the upper and lower pointer are different. There are 8segments and they have a binary address 0-8 essentially. The upperarbiter makes sure that the upper and lower pointer have differentsegment addresses and that the lower pointer is leading by at least onefull buffer. As soon as the lower pointer leads the upper pointer by atleast one full buffer the protocol tells the upper arbiter that it isclear to write. At that point a packet of information is passed to theupper control machine 110, that tells the upper control machine the pipenumber and the command data.

The upper control machine 110, is a state machine illustrated in FIG. 15which generally functions like the lower control machine discussedhereinbefore. Initially, it is waiting 138 for the command and pipeinformation to be passed to it from the upper arbiter 108. When itreceives information from the upper arbiter it places 140 theinformation on a bus to the upper machine 50. Thereafter it asserts 142a signal UMGO telling the upper machine to go, and then waits 144 for anacknowledge, after which the upper control machine will send a releaseflag back to the upper arbiter 108.

Additionally, the upper control machine 110 is concerned with globalmemory alignment. The global memory has a natural alignment of 32 doublewords. Every 32 double words or chunks of memory is a natural boundarywhere the lower 5 bits are all zeros. That is, if the lower 5 bits areall zeros in the address register then it is aligned on a natural memoryblock boundary. Because the maximum burst size that goes across thebackplane is 32 double words the only way that a full 32 double wordburst can be done is if the burst starts at the base on a natural memoryboundary. The best utilization of the backplane occurs when as manytransfers as possible occur with a full 32 double word burst. Therefore,the upper control machine performs the necessary calculations to ensurethat happens.

For example, if a transfer starts misalligned, where the first 5 bitsare not zero, the upper control machine 110 will do the necessary mathto figure out how to get itself back onto a boundary. It will transferjust enough data to get itself back onto a boundary and from that pointon to the end of the transfer it will do full bursts of data, in orderto maximize throughput.

The upper control machine 110 also does burst size calculations to tellthe upper machine 50 exactly how much data it is going to be bursting.Also, there is a configuration register that is set on a global level inthe middle machine that can tell the middle machine how many doublewords it is allowed to burst. The upper control machine will take thatinto consideration as well when it does burst size calculations. Once itcalculates the burst size, the upper control machine 110 will incrementthe address pointer by the number of double words that it will tell theupper machine 50 to transfer.

In the data transfer state machine implementation according to theinvention, there are two sets of handshakes. There is one that initiatesthe upper machine 50 transfer and one that terminates with status. Asdiscussed, the upper control machine 110 initiates the upper machine 50transfer. The upper completer 112, a state machine illustrated in FIG.16, functions substantially like the lower completer describedhereinbefore. Initially, it is waiting in an idle state 138 for anindication that the upper control machine 110 has some activity.

The upper completer 112 receives a signal !UMNXT first from the uppercontrol machine 110 and then a packet from the upper machine 50. Thepacket from the upper machine is the information regarding the transferto be put in the register file 100. The upper completer 112 requests 140the register file 100 and when it can access it, it puts 142 theinformation from the upper machine into the register file 100.Thereafter it issues a next acknowledge signal (UMNACK) 144 to the uppermachine to complete the handshaking. The upper completer 112 thenreturns to the idle state 138.

In the process of accessing register file information, the informationis checked to effect operations indicated by the nature of the bits setas described hereinbefore with respect to the register file contents.Upon completion, the packet of information that is transferred betweenthe upper machine and the middle machine contains the status and thepointers. The upper machine reports back to the middle machinerequesting pointer information. The upper machine captures errors thatoccurred while the transfer is in progress. There are protocol errorsthat can be reported if the upper machine detects something in error intransfer protocol. The upper machine, like the lower machine, can abortthe transfer and report back status regarding the abort.

The upper completer 112 ultimately receives back a packet of informationfrom the upper machine 50 containing error status information and DPRpointer information and it is responsible for putting the informationback in the register file memory 100 inside the middle machine 48.Therefore, the upper completer 112 puts the pointer into the pointerlocation in the status register. The upper completer will then look atthe error bits. If there were errors that were reported by the uppermachine the upper completer will set the appropriate error flags in thestatus register. At this point in a write operation there is data on thebackplane and it will be transferred to global memory. When the globalmemory returns status information the upper machine 50 captures that andwill send a packet to the upper completer 112.

The lower machine functionality is substantially similar whether you arereading or writing. The upper machine has intelligence built in that ifa read from global memory is being done the transfer length registerindicates how much to read from global memory. A 2 k buffer istransferred on the upper side of the middle machine, thus the uppermachine will prefetch and fill that 2 k of DPR as soon as the commandregister is programmed. On a read from global memory the address, thetransfer length, and how much memory is available in DPR is known. Theupper machine can do a prefetch and fill 8 buffers if it has that muchdata to transfer. Thus the data is in the DPR and ready for the lowermachine to go. As the lower machine starts taking data from the DPR anda buffer becomes available the upper machine will fill it again.Accordingly, the amount of latency is reduced by keeping as much data asis possible in the DPR for read transfers. Similarly, with the statemachine configuration described hereinbefore, bandwidth is increased byhaving these independent simultaneous processing machines. By passingpackets of information from machine to machine the previous stage isfree to do more work.

For DPR coherency purposes, the Middle Machine uses the FP bit in theCommand & Transfer Length Register, along with the Upper and Lowerpointers, to determine the current DPR buffer status during a read orwrite operation. This bit can be used by software to monitor the currentstatus of an operation.

During a read operation, the FP bit will be set by the hardware when theLower Machine side of the hardware (the CPU side) is faster than theUpper Machine side of the hardware (the Global Memory side). If this bitis set, it indicates that there is no data available to be read (bufferempty), unless this is the last transfer.

As described, basic Global Memory read operations are initiated by theMiddle Machine, and begin immediately after the ED and IEC bits are setin the Middle Machine Command Register. In other words, the MiddleMachine will automatically begin prefetching data (up to the programmedtransfer length) as soon as it is enabled. The data will be placed inthe DPR for an ADT or DMA pipe. Initiation of a READ operation requiresthe following steps.

Write ‘0000 000h’ to the Middle Machine Interrupt Status Register toclear any pending interrupts for this processor.

Write ‘0000 000h’ to the Middle Machine Status Register to clear theending status of a previous transfer.

Write a valid 32-bit Dword address to the Middle Machine Primary AddressPointer.

Write ‘xxxx 0C41h’ to the Middle Machine Command Register to initiate aRead Command (where ‘xxxx’ is the size of data to transfer).

If the transfer length is 2K bytes or less, either wait for an interruptor poll the Middle Machine Interrupt Status Register until theappropriate interrupt bit is set. If the transfer length is greater than2K bytes, check the current transfer count in the Middle Machine CommandRegister to determine if data is available to read.

If an interrupt occurred, write ‘0000 0000h’ to the Middle MachineInterrupt Status Register to clear the pending interrupt for thisprocessor.

Read the Middle Machine Status Register to determine ending status ofthe data transfer.

During a write operation, the FP bit will be set by the hardware whenthe Upper Machine side of the hardware is faster than the Lower Machineside. If this bit is set, it indicates that there is not enough data toinitiate a Global Memory backplane cycle (buffer empty). “Fullness” canbe determined by comparing UP0-UP2 and LP0-LP2 in the Middle MachineStatus Register.

Similar to read operations, for write operations the Middle Machine mustbe programmed with a transfer length and the EC and IEC bits set in theMiddle Machine Command Register. However, data movement will notprogress until: (a) at least one of the eight buffer segments fills (viathe ADT or DMA pipes), or (b) an EOT is encountered. The middle Machinewill decrement the transfer counter with each Global Memory transfer,and will automatically stop when the count reaches zero. Furthermore, ifan EOT is detected from the ADT or DMA pipe and the transfer count isnot zero, the Middle Machine will post an Ending Transfer Count Not Zeroerror in the Status Register (ETNZ bit). Initiation of a WRITE operationrequires the following steps.

Write ‘0000 000h’ to the Middle Machine Interrupt Status Register toclear any pending interrupt for this processor.

Write ‘0000 0000h’ to the Middle Machine Status Register to clear theending status of a previous transfer.

Write a valid 32-bit Dword address to the Middle Machine Primary AddressPointer.

Write ‘xxxx 0C01h’ to the Middle Machine command Register to initiate aWrite Command (where ‘xxxx’ is the size of the data to transfer).

Either wait for an interrupt or poll the Middle Machine Interrupt StatusRegister until the appropriate interrupt bit is set.

Write ‘0000 0000h’ to the Middle Machine Interrupt Status Register toclear the pending interrupt for this processor.

Read the Middle Machine Status Register to determine ending status ofthe data transfer.

The Middle Machine contains hardware which permits the same data to bewritten automatically to two separate Global Memory addresses, anoperation referred to as a “mirrored write”. The hardware will performtwo backplane write bursts, the first to the Primary Address and thesecond immediately following to the Mirror Address. The remainder of thewrite operations remains the same as described hereinabove. For MirrorWrites to function properly, the Primary and Mirror Addresses must be onthe same Global Memory alignment (the lower 5 bits of the Primary andMirror Addressed must be the same). If not a count miss will occur, Toinitiate a Mirror WRITE operation, follow these steps:

Write ‘0000 0000h’ to the Middle Machine Interrupt Status Register toclear any pending interrupts for this processor.

Write ‘0000 0000h’ to the Middle Machine Status Register to clear theending status of a previous transfer.

Write a valid 32-bit Dword address to the Middle Machine Primary AddressPointer.

Write a valid 32-bit Dword address to the Middle Machine Mirror/CopyAddress Pointer.

Write ‘xxxx 0C81h’ to the Middle Machine Command Register to initiate aMirror Write Command (where ‘xxxx’ is the size of data to transfer).

Either wait for an interrupt or poll the Middle Machine Interrupt StatusRegister until the appropriate bit is set.

If polling, wait until the DMC bit or the Middle Machine Status registeris set.

Write ‘0000 000h’ to the Middle Machine Interrupt Status Register toclear the pending interrupt for this processor.

Read the Middle Machine Status Register to determine ending status ofthe data transfer.

The Middle Machine can also be programmed to operate as a true DMA(Direct Memory Access) machine. In this mode, the hardware can performdirect memory transfers from any Global Memory Address to any otherGlobal Memory Address. Given a transfer Length register of 14 bits, thehardware can transfer up to 16 k Memory Dwords at a line (128 k ofdata). This function may be useful, for example, to perform block copiesof data from one memory region to another or to quickly initializememory (by copying an initialized block of memory to other regions). Aswith Mirror Writes, for Copy to function properly, the Primary and Copyaddresses must be on the same Global Memory alignment (the lower 5 bitsof the Primary and Copy Addresses must be the same). If not, a countmiss will occur. To initiate a COPY operation, follow these steps:

Write ‘0000 0000h’ to the Middle Machine Interrupt Status Register toclear any pending interrupts for this processor.

Write ‘0000 0000h’ to the Middle Machine Status Register to clear theending status of a previous transfer.

Write a valid 32-bit Dword address to the Middle Machine Primary AddressPointer. This becomes the Source (READ) Address.

Write a valid 32-bit Dword address to the Middle Machine Mirror/CopyAddress Pointer. This becomes the Destination (WRITE) Address.

Write ‘xxxx 0C45h’ to the Middle Machine Command Register to initiate aMirror Write Command (where ‘xxxx’ is the size of data to copy). Itshould be noted that the RW bit must be set for Copy to operatecorrectly.

Write ‘0000 000h’ to the Middle Machine Interrupt Status Register toclear the pending interrupt for this processor.

Read the Middle Machine Status Register to determine ending status ofthe data transfer.

Referring again to FIG. 8, in addition to the “round robin” arbitrationthat is effected in the data transfer state machines describedhereinbefore, a line processor arbitration mechanism, the CPU arbiter,146 is provided in the Middle Machine 48. The CPU arbiter 146 interfacesto the lower arbiter 102, and basically arbitrates among the lineprocessors and service processor for access to the data transfer statemachines.

A CPU I/O (input/output port) 148 is also configured into the lower sideto effect direct processor access to the DPR and/or global memorythrough the middle machine 48.

Service Processor and Messaging

Referring back to FIG. 3, the shared resources portion of the ESCONinterface according to the invention includes the service processor 70in addition to the line processors 60. The service processor 70 is a“shared resource” that is responsible for starting the operation(s) ofthe ESCON interface at the beginning of time when the ESCON interface isfirst powered on. The service processor 70 prepares the board, loads allthe FPGAs, brings up the board, runs diagnostics, and then once it hasdetermined that everything is functional it will turn on the lineprocessors. The service processor is a high end, low power, highperformance and low cost processor with the ability to be interfacedinto a system, such as a power PC 603E or the like.

The service processor 70 also performs system wide services. When thereis a system wide interrupt it is delivered to the service processor 70,which acts upon the interrupt. The service processor 70 determineswhether it is appropriate to interrupt the data transfer that is inprogress or not.

The service processor 70 has a Small Computer System Interface (SCSI)port. In the case of a power failure the service processor 70 shuts downthe ESCON line(s) and then dumps the data to disk.

The service processor 70 is further responsible for consolidating someof the operations from each of the line interfaces/processors 60. If allof the line processors performed all of the accesses to the backplanethemselves there would be numerous accesses. The line processors deferand consolidate their actions onto the service processor 70 so that theservice processor can do the operation on behalf of the whole card. Iftwo line processors were to access memory it would require twice thebandwidth than if the service processor accessed memory on behalf ofthem.

The service processor 70 provides a central communications point betweenall the line processors in the rest of the system. A messaging system,as illustrated in FIG. 17, is implemented according to the invention toget the messages from the line processors 60 to the service processor,to tell it the service processor that at least one of the lines has anoperation that has to be performed. Similarly, the messaging mechanismis used to return values or give information to the line processors.

The messaging mechanism uses shared, low latency, Static RAM 150 ineffect as a mailbox, accessible to the line processors 60 and theservice processor 70 over the shared bus. The shared SRAM 150 is dividedinto “mailboxes” 152, i.e. addresses or address ranges. Some of theaddresses are available for reading and writing (inbound and outbound)by the service processor 70 and reading by a respective line processor.Other addresses in the shared SRAM are available for writing by therespective line processor(s) 60 and reading by the service processor 70.

The service processor and all the line processors have access to theshared SRAM 150. There is a secondary mechanism on the power PC, used asline and service processors, in which each line processor has aninterrupt, called a system management interrupt. These system managementinterrupts are used to create a simple messaging arrangement between theservice processor 70 and the line processors 60. The mailboxes in memoryare effectively reserved for each respective processor, so that theservice processor can put a message for a particular line processor intoits respective mailbox 152. Once the service processor 70 writes themessage in the mailbox 152 it pulls the interrupt line(s) on theapplicable line processor(s).

The respective line processor then receives the interrupt, goes to thecode for the interrupt vector, which is an instruction to retrieve themessage and decode it. The line processor decodes the message andperforms the requested operation.

The line processor can then put a result back into another memorylocation which is reserved for the service processor, and then pull theinterrupt to the service processor. The service processor retrieves itsmessage which would be the return codes of whatever operation wasperformed. This messaging scheme saves the processors the trouble ofhaving to poll the shared memory all the time to see if there is amessage in its mailbox location. However, it should be appreciated thatmailboxes can be dedicated for writing (inbound) by the line processors60 and be polled periodically by the service processor 70 (outbound).

A primary function of this messaging system is when the ICDA is loadedwith traffic, i.e. requests to memory, the service processor can bebasically used as a consolidator to condense the operations. In theillustrative ICDA, when the host makes a request for data the lineprocessor looks to see if it is in global memory or whether it is on thedisc. When the line processor is looking to see if it is in globalmemory it is a very fast operation. There are 2 DMA pipes coming fromeach line, a high priority and a low priority. The line processor willgo down the high priority line to see if the data it is looking for isin global memory. If the data is in global memory then the lineprocessor will perform the data transfer autonomously.

However, if the data is not in global memory then the line processor hasa second option depending on how the ICDA is loaded. That is, when oneof the line processors gets a request from the host it looks in globalmemory for the data. If the data requested is not in global memory itmay be desirable for the line processor to give the service processorthe message to go get the data off the disc while the line processordoes other things. Therefore, the line processor sets up an instructionin the service processor's mailbox in shared memory that in effect is arequest to go get data from the disc drive. The instruction issued bythe requesting line processor to the service processor provides anindication of the data that is requested, and other information thataccompanies a data request.

The line processor then sends the service processor an interrupt toalert it to check the shared memory for the message. Meanwhile the lineprocessor is going to go through the operation of disconnecting from thehost and preparing for another connection. The line processor might getanother connection and start performing other transfers. If the nextdata that is requested is not in global memory then the line processormay again defer to the service processor as described, via the messagingmechanism. All of the line processors are doing this so there will be alist of requests which the service processor will consolidate.

The software running on the service processor can be configured todetermine how best to consolidate the requests. If some of the requestsare for data that has contiguous locations, or is from the same disccard or is just closer to other data being retrieved the serviceprocessor can determine the best order to process the requests tooptimize system performance.

The messaging scheme can be implemented to give priority to desiredoperations to enhance overall system performance. A pure data requestcoming from the host machine would have the ultimate priority to enhancethroughput. That data would be delivered immediately to the globalmemory. The secondary priority could be given to the service processorto perform its interfaces with the global memory because the serviceprocessor transfers are performed on behalf of the four processors. Laston the priority list would be each of the line processors going to discfor information directly, because it would only compromise theperformance on one line for a brief period of time.

If the line processor has not performed an operation in a long time, setby a software timer, and it appears that the ICDA is lightly loaded, theline processor can decide that it is going to handle the operationitself even though the data is not in global memory. The line processordoes not defer to the service processor because that would just addadditional delay. Because the line processor was not busy it would notsave time to consolidate the operation to the service processor. ABoolean flag can be set in the software telling the line processor tojust do everything itself, and it will use the low priority line to goto the memory. Under the low priority request into the global memory ifthere is a higher priority request the line processor will have to wait.

The Global Memory in the referenced ICDA includes a lock feature tofacilitate read and write operations. It is designed to prevent theoverwriting of data by more than one line interface accessing globalmemory at the same time. When a lock is asserted on one of the ports,the memory board will not allow access through the other ports until thelock condition is cleared again.

In order to put these requests into a queue it is necessary to lock theglobal memory while the queue and the pointer that points to the end ofthe queue are updated. Because the memory is a shared resource at thatpoint it must be locked to prevent an overwrite of data. The serviceprocessor can lock the memory, write all of the requests and then unlockthe memory. It is a shorter operation then if each one of the lineprocessors locked the memory and wrote the data, because there is anoverhead associated with each operation of locking and unlocking. If theservice processor can perform the transfer on behalf of all the lineprocessors it is quicker than each operation individually. If theservice processor delivers several requests at one time it can burst thedata into memory.

As described, when a selected threshold of activity is reached in theICDA the service processor invokes the messaging system at a point thatefficiency is gained by consolidating. Additionally, the messagingmechanism also gets invoked for other things as well, for instance todisplay anything on a console connected to the central point ofcommunication, i.e. service processor. The service processor is the onlyone that is attached to the outside world so it is the only one that isattached to the terminal. If a line processor for instance detects afault or an error the only way it can report that to the world is totell the service processor to display it on the console.

The line processor with the error to display passes a message to theservice processor that requests the display and indicates the messagefor the console. The service processor will display it on the console,while sending the message to other shared resources for data logging.

Although the processors described herein are power PC's (MPC603E-66/100)one skilled in the art would appreciate that other processors such asIntel Pentium, Motorola or other processors could be used.

Although the adapter architecture is described herein as receiving datafrom a host with ESCON I/O, it will be appreciated the architecturecould be implemented generally in a SCSI or parallel channel adapter, orthe like.

Although the enhanced parity is described herein with respect to FIFOintegrity in the context of a data transfer pipeline in an ESCONinterface for an ICDA, it will be appreciated by one skilled in the artthat it could be applicable other contexts as well.

Although the messaging mechanism described herein is implemented withrespect to an ESCON adapter it willed be appreciated that it could beimplemented in other systems, such as on an SCSI or fiber channeladapter.

Although the lower, middle and upper machines described herein areimplemented in a FPGA (field programmable gate array) it willed beappreciated that other programmable devices, large scale integratedcircuits and/or application specific integrated circuits could be usedto implement the functionality described herein.

Similarly, while the functionality described herein in the variouscomponents is described as implemented primarily in hardware, it will beappreciated that such functionality can be implemented as well insoftware modules, as any of various hardware implementations asdescribed above, and/or as combinations of hardware, software andfirmware.

Although the invention has been shown and described with respect toexemplary embodiments thereof, various other changes, omissions andadditions in the form and detail thereof may be made therein withoutdeparting from the spirit and scope of the invention.

1. An apparatus for connecting a host Enterprise System ConnectionArchitecture (ESCON) Input/Output (I/O) interface to a cache of a datastorage system, the apparatus comprising: a set of at least 4 pipelines,each pipeline being coupled on a first end to the host ESCON I/Ointerface and being coupled on a second end to the cache; a plurality ofline processors, each line processor controlling one or more of thepipelines of the set of pipelines; and in each pipeline, a protocolengine, the protocol engine configured to distinguish user data fromframe header data and separate the user data from the frame header datafor transport over the pipeline; wherein: the apparatus furthercomprises a plurality of assembler/disassembler mechanisms, eachassembler/disassembler mechanism coupling to one or more pipelines; eachpipeline further comprises: a first first-in first-out (FIFO) buffer,wherein the separated user data from the protocol engine is temporarilystored in the first FIFO buffer before being transmitted to anassembler/disassembler mechanism in a first direction for a writeoperation; a second FIFO buffer, configured to receive additional datafrom the assembler/disassembler mechanism and to transmit the additionaldata to the protocol engine in a second, opposite, direction for a readoperation; a receive frame memory, configured to receive frame headerdata from the protocol engine; and a transmit frame memory, configuredto transmit frame header data to the protocol engine; the protocolengine is further configured to compute error detection information forthe user data; the user data is stored in the first FIFO buffer togetherwith the error detection information; and each assembler/disassemblermechanism is configured to: receive user data from the first FIFObuffer; generate error detection information for the user data; andoutput, based on the generated error detection information and the errordetection information stored in the first FIFO buffer, a signalindicating whether the user data has been received in proper sequencefrom the first FIFO buffer.
 2. An apparatus as in claim 1 wherein eachassembler/disassembler mechanism is further configured to accumulateuser data until a threshold quantity of data is accumulated for anefficient transfer and subsequently pass the accumulated data along thepipeline in a single burst.
 3. A method for connecting a host EnterpriseSystem Connection Architecture (ESCON) Input/Output (I/O) interface to acache of a data storage system, the method comprising: connecting thehost ESCON I/O interface to the cache over at least 4 pipelines, eachpipeline being controlled by one of a plurality of line processors;placing in each pipeline a protocol engine for distinguishing user datafrom frame header data; separating the user data from the frame headerdata, at a protocol engine of the plurality of protocol engines, fortransport over the pipeline; calculating error detection information forthe user data at a protocol engine of the plurality of protocol engines;receiving, at a receive frame memory, frame header data from theprotocol engine; temporarily storing the separated user data for a writeoperation together with the error detection information in a firstfirst-in first-out (FIFO) buffer associated with the pipeline;receiving, in a first direction, at an assembler/disassembler mechanism,the separated user data from the first FIFO buffer; generating, at theassembler/disassembler mechanism, error detection information for theseparated user data; outputting, based on the generated error detectioninformation and the error detection information stored in the first FIFObuffer, a signal indicating whether the user data has been received inproper sequence from the first FIFO buffer; transmitting, from atransmit frame memory, frame header data to the protocol engine; andtransmitting, for a read operation in a second, opposite, direction,from a second FIFO buffer, additional data to the protocol engine.
 4. Amethod as in claim 3 wherein the method further comprises, at theassembler/disassembler mechanism, accumulating user data until athreshold quantity of data is accumulated for an efficient transfer andsubsequently passing the accumulated data along the pipeline in a singleburst.